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ABSTRACT 


In  the  last  20  years,  the  Department  of  Defense  has  seen  a  general  reduction  in 
the  acquisition  workforce  yet  experienced  unprecedented  growth  in  spending  for 
services.  The  government  has  increased  the  level  of  scrutiny  on  service 
contracts  highlighting  the  need  to  consistently  follow  policy  when  documenting 
contractor  performance  in  the  Contractor  Performance  Assessment  Reporting 
System  (CPARS). 

This  report  used  statistical  analysis  to  examine  715  Army  service 
contractor  performance  reports  in  CPARS  in  order  to  answer  the  following 
questions:  (1)  Are  government  contracting  professionals  submitting  contractor 
performance  narratives  in  accordance  with  CPARS  guidelines?  (2)  What  is  the 
added  value  of  the  contractor  performance  narratives  beyond  the  value  of  the 
objective  scores  for  performance?  (3)  What  is  the  statistical  relationship  between 
the  narratives  and  the  objective  scores?  (4)  Do  the  interview  findings  contradict, 
support,  or  enhance  the  findings  for  the  three  questions  above?  (5)  What 
conclusions  or  recommendations  can  we  draw  from  the  answers  to  the  questions 
above? 

The  findings  revealed  inconsistencies  between  the  objective  scores  and 
narratives  entered  into  CPARS.  This  report  discusses  the  findings  and  concludes 
with  eight  recommendations  to  improve  documenting  contractor  performance. 
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I.  INTRODUCTION 


A.  BACKGROUND 

The  purpose  of  this  chapter  is  to  introduce  the  system  designated  by  the 
Department  of  Defense  (DOD)  for  reporting  contractor  past  performance,  called 
Contractor  Performance  Assessment  Reporting  System  (CPARS).  The  DOD 
obligated  approximately  $360  billion  in  fiscal  year  2012  for  contracts  for  supplies 
and  services  (Government  Accountability  Office  [GAO],  2013).  In  order  to 
support  best  practices  for  government  acquisition,  the  Office  of  Federal 
Procurement  Policy  (OFPP)  directs  in  Federal  Acquisition  Regulation  (FAR)  Part 
42  that  federal  contracting  agencies  shall  use  CPARS  as  their  primary  tool  for 
documenting  contractor  past  performance. 

This  MBA  project  focuses  on  Army  services  contracts,  specifically,  the 
narrative  section  of  contractor  performance  assessment  reports,  and  advances 
the  research  conducted  by  Hart,  Stover,  and  Wilhite  (2013),  which  was  focused 
on  the  CPARS  contractor  report  cards.  This  research  project  generated  a  local 
database  of  715  Army  service  contracts  that  were  used  to  correlate  the  success 
of  contracts  based  on  objective  scores  of  six  characteristics. 

This  is  the  eighth  research  project  in  a  series  of  MBA  projects  relating  to 
services  acquisitions.  Apte,  Dixon,  and  Rendon  are  collaborating  on  research 
that  focuses  on  the  value  of  CPARS  narratives,  the  correlation  of  narratives  to 
objective  scores,  and  the  narratives’  effect  upon  the  source  selection  process. 

Government  contracting  professionals  use  contractor  past  performance 
information  (PPI)  during  source  selection  and  contract  administration.  The  FAR 
requires  that  contracting  officers  consider  contractor  past  performance  as  one 
evaluation  factor  for  awarding  contracts.  Contractor  PPI  can  be  the  factor  that 
determines  whether  a  particular  contractor  will  be  awarded  government  business. 
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B.  PROBLEM  STATEMENT 


CPARS  is  a  system  that  was  designed  to  be  used  by  contracting 
professionals  for  submitting  and  retrieving  contractor  past  performance 
information.  There  have  been  many  discrepancies  and  failures  specified  in 
recent  Government  Accountability  Office  (GAO)  reports  with  the  utilization  of 
CPARS.  Government  contracting  agencies  are  failing  to  properly  document 
contractor  PPI  within  the  schedule  requirements  mandated  by  the  OFPP.  In  a 
2009  report,  data  analyzed  from  2007  showed  that  DOD  contracting  components 
completed  required  contractor  past  performance  report  cards  less  than  half  of  the 
time  (GAO,  2009).  Subsequently,  the  DOD  increased  its  focus  on  training  and 
education  for  contracting  professionals,  which  led  to  an  increase  in  contractor 
performance  assessments  being  completed  and  submitted  to  the  Past 
Performance  Information  Retrieval  System  (PPIRS).  In  the  last  published  report 
during  the  research  period  of  this  project  (GAO,  2013),  the  GAO  noted  significant 
gains  in  completion  rates.  Fifty-six  percent  of  required  reports  were  completed  in 
2011,  while  74%  were  completed  in  2013.  Flowever,  according  to  the  same 
report,  over  half  of  these  reports  were  submitted  late.  In  addition,  many  CPARS 
report  cards  contain  narratives  that  are  either  insufficiently  detailed  or  conflict 
with  their  associated  objective  scores.  Late  reports  lacking  sufficient  accurate 
information  provide  less-than-optimal  information  to  the  contracting  professionals 
who  rely  on  these  report  cards  for  source  selection  and  contract  administration 
purposes. 

C.  PURPOSE 

Government  contracting  professionals  submit  to  CPARS  information  on 
contractors  related  to  their  contract  performance.  This  information  is  in  the  form  of 
objective  scores  in  five  categories:  quality,  schedule,  cost  control,  business 
relations,  and  management  of  key  personnel.  In  addition  to  these  five  categories, 
there  is  a  specific  contractor  evaluation  section  in  which  the  government  evaluator 
writes  a  descriptive  narrative  of  the  contractor’s  performance. 
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The  purpose  of  this  research  is  to  attempt  to  determine  the  value  of 
contractor  performance  assessment  report  narratives  for  services  contracts  by 
comparing  the  relationships  between  narratives  and  objective  scores  in  order  to 
recommend  improvements  to  the  CPARS  contractor  performance  information 
documentation  process.  These  recommendations  regarding  the  process  aim  to 
improve  the  CPARS  report  card  product,  which  should  lead  to  greater  and  more 
effective  utilization  of  the  CPARS  system  for  source  selection  and  contract 
administration  purposes. 

D.  RESEARCH  QUESTIONS 

The  primary  question  addressed  by  this  research  project  is  as  follows: 
Does  the  CPARS  report  card  written  narrative  section  provide  value  to  the 
contractor  performance  evaluation  process?  This  research  focuses  on  five 
research  questions  that  explore  the  nature  of  CPARS  narratives: 

1 .  To  what  degree  are  government  contracting  professionals 
submitting  contractor  performance  narratives  to  CPARS  in 
accordance  with  the  guidelines  provided  in  the  CPARS  user’s 
manual? 

2.  What  is  the  added  value  of  the  contractor  performance  narratives 
beyond  the  value  of  the  objective  scores  for  performance? 

3.  What  is  the  statistical  relationship  between  the  sentiment  contained 
in  the  narratives  and  the  objective  scores  for  contractor 
evaluations? 

4.  To  what  degree  do  the  interview  findings  contradict,  support,  or 
enhance  the  findings  for  the  three  research  questions  above? 

5.  What  conclusions  or  recommendations  can  we  draw  from  the 
answers  to  the  research  questions  above? 
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E.  METHODOLOGY 

This  research  examines  the  value  of  CPARS  report  card  narratives  for 
service  contracts  as  they  relate  to  their  associated  objective  scores.  The 
methodology  includes  a  literature  review,  data  analysis  (including  both  sentiment 
and  statistical  analysis),  and  interviews  with  government  agency  contracting 
professionals. 

The  literature  review  was  conducted  for  the  purpose  of  examining  and 
considering  knowledge  available  from  the  existing  body  of  literature  on  the 
subjects  of  federal  contracting  and  government  contractor  performance.  The 
GAO  and  OFPP  have  released  several  studies  and  memoranda  on  the  subject  of 
contractor  past  performance  as  it  relates  to  the  government  contracting  process. 
This  research  was  based  in  large  part  upon  these  documents  and  previous 
research  performed  by  Hart,  Stover,  and  Wilhite  (2013)  that  explored  the  drivers 
of  successful  service  contracts.  Building  on  their  research,  this  project 
conducted  a  review  of  literature  about  the  service  contract  management  process 
and  the  Contractor  Performance  Assessment  Reporting  System. 

The  first  phase  of  data  analysis  performed  was  a  sentiment  analysis  of 
715  Army  service  contract  CPARS  report  card  narratives.  The  CPAR  Quality 
Checklist  was  used  as  a  basis  for  developing  the  criteria  for  the  categories  and 
values  for  the  analysis  {CPAR  Quality  Checklist,  n.d.).  Narratives  were  assigned 
several  scores  relating  to  their  quality,  robustness,  compliance  with  directions  in 
the  CPAR  Quality  Checklist,  and  their  value  and  content  compared  to  their 
related  objective  scores  from  the  CPARS  report  cards. 

The  second  phase  of  data  analysis  included  a  statistical  analysis  of  the 
relationship  between  the  scores  provided  by  the  sentiment  analysis  and  their 
associated  objective  scores.  This  analysis  looked  at  correlating  relationships 
between  the  various  sentiment  scores  and  the  various  objective  scores  for  the 
same  CPARS  report  to  find  meaningful  relationships  between  them  and  to  reveal 
the  extent  of  the  value  of  the  narratives. 
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In  support  of  the  data  analysis,  interviews  with  contracting  professionals 
from  two  DOD  contracting  agencies  were  conducted.  These  interviews  consisted 
of  a  series  of  questions  asking  the  subjects  about  how  they  use  and  to  what 
extent  they  value  CPARS  and  other  sources  of  contractor  past  performance 
information  and  to  what  extent  they  value  the  narratives  compared  to  the 
performance  objective  scores. 

F.  BENEFITS  AND  LIMITATIONS 

The  purpose  of  this  research  is  to  attempt  to  determine  the  value  of 
contractor  performance  assessment  report  narratives  for  services  contracts  by 
comparing  the  relationships  between  narratives  and  objective  scores.  Potential 
benefits  to  this  research  include  a  better  understanding  of  the  value  of  the 
CPARS  narratives  and  the  extent  to  which  DOD  contracting  professionals  value 
them  relative  to  other  sources  of  contractor  past  performance  information.  This 
study  will  provide  information  based  on  data  indicating  how  closely  written 
narratives  correlate  to  their  associated  objective  scores.  It  will  also  show  how 
well  DOD  approving  officials  and  the  individuals  contributing  to  the  narratives 
comply  with  the  directions  for  completing  a  CPARS  narrative  as  instructed  in  the 
CPAR  Quality  Checklist  {CPAR  Quality  Checklist,  n.d.). 

There  are  clearly  identifiable  limitations  to  the  research  being  conducted. 
First,  this  research  project  is  limited  to  the  715  Army  service  contracts  analyzed 
in  a  previous  Naval  Postgraduate  School  (NPS)  MBA  project  (Hart  et  al.,  2013). 
The  selection  criteria  for  those  715  contracts  was  limited  by  Army  service 
contracts,  “contracts  only  submitted  by  Mission  Installation  Contracting 
Commands  (MICCs),  and  contracts  from  only  five  MICC  offices”  (from  Hart  et  al., 
2013,  p.  5). 
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The  next  limitation  is  the  dollar  threshold  for  reporting  contractor 
performance  information  in  CPARS.  DOD  CPARS  policy  provides  different  dollar 
thresholds  for  mandatory  reporting  of  contractor  past  performance  (PPI) 
information.  Service  contracts  must  report  contractor  PPI  starting  at  dollar 
amounts  greater  than  $1,000,000  (Department  of  the  Navy  [DON],  1997).  This 
means  contractor  PPI  for  services  contracts  may  not  be  reported  if  dollar 
amounts  are  below  $1  million.  This  limits  the  research  database  to  either  service 
contracts  above  $1  million  or  service  contracts  below  the  threshold  that  DOD 
contracting  professionals  chose  to  enter  into  CPARS. 

Another  limitation  of  this  research  is  the  fact  that  DOD  contracting 
professionals  may  not  enter  PPI  into  CPARS  even  though  it  is  required.  As  of 
the  completion  of  this  project,  the  most  recent  report  (GAO,  2013)  shows  this 
lack  of  performance  with  56%  and  74%  of  CPARS  reports  completed  in  201 1  and 
2013,  respectively.  These  limitations  are  discussed  further  in  later  chapters. 

G.  SCOPE  AND  ORGANIZATION  OF  RESEARCH 

This  research  report  consists  of  five  chapters  organized  in  the  following 
manner.  Chapter  I  contains  background  information,  a  problem  statement,  the 
purpose  of  the  research  project,  research  questions,  the  methodology  of 
research,  the  benefits  and  limitations  of  the  project,  and  the  scope  and 
organization  of  the  research  project.  Chapter  II  is  a  literature  review  that  provides 
an  overview  of  the  DCD  acquisition  process  for  service  contracts,  the  contractor 
performance  assessment  reporting  system,  the  increase  in  scrutiny  of  federal 
contracting  agencies,  and  previous  studies  that  led  us  to  this  research.  Chapter 
III  contains  the  methodology  for  how  the  research  was  performed,  organized,  and 
analyzed.  This  chapter  also  discusses  the  purpose  and  scope  of  interview 
questions  posed  to  DCD  contracting  professionals.  Chapter  IV  provides  an 
analysis  of  the  data  and  presents  the  answers  to  the  research  questions. 
Chapter  V  summarizes  the  research  project,  provides  conclusions,  and 
addresses  potential  areas  for  additional  research. 
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H.  SUMMARY 


This  chapter  provided  the  background  information  leading  up  to  the 
research  project,  a  problem  statement,  the  purpose  of  the  research  project, 
research  questions,  the  research  methodology  used  to  analyze  the  data,  benefits 
and  limitations  of  the  research,  and  the  scope  and  organization  of  the  overall 
project.  Chapter  II  provides  the  literature  review  of  the  DOD  acquisition  services 
contracting  process,  the  CPARS  evaluation  system,  and  previous  studies 
conducted  by  NPS  in  relation  to  services  contracts. 
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II.  LITERATURE  REVIEW 


A.  INTRODUCTION 

The  purpose  of  this  chapter  is  to  provide  an  introduction  to  the  DOD 
acquisition  process  for  services,  with  a  focus  on  Army  services  contracts.  This 
review  attempts  to  familiarize  the  reader  with  the  necessary  information  to 
understand  the  contracting  structure,  contracting  terms,  roles,  and 
responsibilities  of  the  acquisition  team,  and  the  system  that  the  DOD  has 
implemented  for  documenting  and  retrieving  contractor  past  performance 
information,  called  the  Contractor  Performance  Assessment  Reporting  System 
(CPARS).  This  chapter  provides  a  discussion  about  the  purpose  of  CPARS  as 
well  as  current  issues  associated  with  CPARS.  The  chapter  includes  a  look  at 
reports  submitted  by  a  number  of  government  authorities  that  point  out  the 
successes  and  failings  of  various  government  contracting  agencies  in  complying 
with  laws  and  directives  specifying  how  and  to  what  extent  they  are  to  utilize 
CPARS  for  documenting  contractor  past  performance  information. 

B.  THE  DOD  ACQUISITION  PROCESS  FOR  SERVICE  CONTRACTS 

To  understand  the  DOD  acquisition  process,  the  reader  first  needs  to 
understand  what  the  term  acquisition  means.  The  Federal  Acquisition 
Regulation  (FAR)  defines  the  term  acquisition  as  follows: 

acquiring  by  contract  with  appropriated  funds  of  supplies  or 
services  (including  construction)  by  and  for  the  use  of  the  Federal 
Government  through  purchase  or  lease,  whether  the  supplies  or 
services  are  already  in  existence  or  must  be  created,  developed, 
demonstrated,  and  evaluated.  Acquisition  begins  at  the  point  when 
agency  needs  are  established  and  includes  the  description  of 
requirements  to  satisfy  agency  needs,  solicitation  and  selection  of 
sources,  award  of  contracts,  contract  financing,  contract 
performance,  contract  administration,  and  those  technical  and 
management  functions  directly  related  to  the  process  of  fulfilling 
agency  needs  by  contract.  (FAR,  2014,  Part  2.101b) 
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The  basic  definition  is  that  the  government  has  a  need  and  fulfills  that  need  by 
purchasing  a  product  or  service  through  a  defined  process  using  a  set  of  rules. 

All  DOD  acquisitions — services  and  non-services  alike — begin  the 
contracting  process  with  the  same  fundamental  step:  identify  a  bona  fide  need  or 
requirement.  The  customer,  usually  an  organization  or  an  entity  within  an 
organization,  identifies  a  product  or  service  that  is  essential  for  that  organization 
to  complete  its  mission  or  function.  The  customer  must  be  specific  when 
describing  the  agency  need  for  the  contracting  officials  to  accurately  identify  the 
requirement  and  begin  the  contracting  process.  According  to  FAR  Part  11, 
Describing  Agency  Needs,  the  customer  must,  to  the  maximum  extent  possible, 
adhere  to  the  following: 

(i)  State  requirements  with  respect  to  an  acquisition  of  supplies  or 

services  in  terms  of  - 

(A)  Functions  to  be  performed; 

(B)  Performance  required;  or 

(C)  Essential  physical  characteristics,  (from  FAR,  2014,  Part 

11.002) 

An  example  of  a  bona  fide  need  is  as  follows.  The  Army  Mission  and 
Installation  Contracting  Command  (AM ICC)  received  a  request  from  a  customer 
to  purchase  janitorial  services.  The  customer  has  funding  available  in  the  current 
fiscal  year  and  is  requesting  12  months  of  services  for  one  building  with  a 
requested  contract  start  date  two  months  from  the  submission  of  the 
requirement.  The  services  consist  of  the  following  tasks:  sweep  floors  daily,  mop 
floors  with  soap  and  water  one  time  per  week,  wax  floors  one  time  per  quarter, 
remove  trash  from  garbage  bins  every  Thursday,  clean  and  sanitize  bathrooms 
daily,  and  other  specified  tasks.  The  service  requirement  is  clearly  defined  with 
funding  available  to  obligate  expenses  and  provides  the  contracting  official  with 
sufficient  information  to  begin  the  acquisition  process,  and  so  a  bona  fide  need  is 
apparent. 
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The  following  request  for  maintenance  services  would  be  rejected  due  to 
its  not  meeting  the  bona  fide  need  requirement.  The  AMICC  receives  a  request 
for  M1A1  Abrams  tank  maintenance  services  from  Fort  Drum’s  10th  Mountain 
Division.  This  request  would  be  denied  as  the  10th  Mountain  does  not  have 
tanks  on  its  Table  of  Organization  and  Equipment  (TO&E).  A  unit  cannot  request 
maintenance  services  for  equipment  it  does  not  have.  Alternatively,  a  request  by 
the  U.S.  Army  Armor  School  at  Fort  Knox  might  have  a  bona  fide  need  for  tank 
maintenance  services  because  they  train  Army  and  Marine  Corps  personnel  on 
the  Abrams  tank. 

Once  a  bona  fide  need  has  been  identified,  the  service  acquisition  process 
can  begin.  The  service  acquisition  plan  is  a  comprehensive  strategy  that 
identifies  the  relationship  between  the  plan’s  phases  and  the  work  to  be 
performed,  lists  the  milestones  to  be  completed,  and  provides  the  overall 
approach  that  management  will  follow  to  mitigate  risk  while  meeting  the  service 
requirements  (Neuman,  2013b). 

The  service  acquisition  process  can  be  identified  by  its  three  phases: 
Plan,  Develop,  and  Execute.  These  phases  break  down  into  seven  distinct 
processes  that  build  upon  the  previous  process.  As  Beers  (2011)  wrote,  “The 
Planning  phase,  as  depicted  in  Steps  1,  2,  and  3,  lays  the  foundation  for  the 
services  acquisition”  (from  Beers,  2011,  p.  56).  According  to  Beers  (2011), 
“During  the  development  phase — Steps  4  and  5 — a  requirements  roadmap 
process  is  used  to  define  performance  objectives  and  standards,  allowable 
variations,  and  method  of  performance  assessment.  In  the  execution  phase — 
Steps  6  and  7 — the  team  puts  all  the  customer’s  planning  and  development 
efforts  into  action”  (from  Beers,  2011,  p.  57).  Figure  1  explains  the  seven  stages 
of  the  service  acquisition  process. 
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•  Select  Ri|ht  Contnctor 

•  Award  Contract 

•  Roll  Out  Strategy 


Execute 

Figure  1.  The  Services  Acquisition  Plan  (from  Beers,  2011,  p.  56) 


Phase  1:  Plan 

The  first  phase  of  the  services  acquisition  process  is  the  planning  phase. 
This  phase  is  extremely  important  as  it  is  the  foundation  for  the  service 
acquisition.  The  planning  phase  contains  three  steps:  (1)  form  the  team, 
(2)  review  current  strategy,  and  (3)  market  research. 

Step  1:  Form  the  Team 

During  Step  1  of  this  phase,  the  agency  head  is  responsible  for 
designating  who  will  be  a  member  of  the  acquisition  team,  provide  them  with  the 
guidance  on  how  to  proceed,  and  ensuring  that  the  plan  complies  with  FAR 
requirements  (FAR,  2014,  Part  7.103[i]).  The  acquisition  team  consists  of  anyone 
who  will  be  responsible  for  significant  portions  of  the  service  acquisition 
throughout  its  life  cycle  (FAR,  2014,  Part  7.102[b]).  No  single  person  has  the 
requisite  knowledge,  technical  proficiency,  time,  or  resources  to  perform  all  the 
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functions  necessary  to  plan,  source,  award  and  administer  a  service  contract,  but 
an  acquisition  team  made  up  of  experts  in  functional  areas  can.  Acquisition  team 
members  can  include,  but  are  not  limited  to,  the  program  manager,  contracting 
officer,  contracting  officer’s  representative  (COR),  and  others  with  expert 
functional  area  knowledge  like  finance,  legal,  engineering,  small  business 
administration,  quality  assurance,  logisticians  and  the  customer/end  user  (DOD, 
2012). 

Program  Manager:  The  program  manager  (PM)  is  the  single  person 
tasked  with  the  responsibility  to  manage  the  acquisition  team  while  balancing  the 
acquisition  cost,  schedule,  and  performance  for  that  program  (Neuman,  2013a). 
PMs  perform  essential  functions  such  as  attending  program  reviews,  defending 
the  program  or  fighting  for  additional  funding,  fighting  requirements  creep,  and 
coordinating  with  senior  DOD  and  congressional  staffers  to  ensure  compliance 
with  statutory  requirements  and  congressional  intent.  Given  the  importance  of 
the  PM  position,  it  is  unfortunate  that  previous  research  by  Apte  and  Rendon 
(2007)  found  that  “in  many  service  acquisitions,  a  project  manager  or  project 
team  is  not  assigned  and  the  contracting  officer  assumes  the  responsibilities  of 
the  project  manager  or  project  team  leader”  (from  Hart  et  al.,  2013,  p.  10). 

Contracting  Officer:  The  contracting  officer  (KO),  as  defined  by  Neuman 
(2013b),  is: 

responsible  for  providing  contracting  support  required  for  meeting 
the  project’s  cost,  schedule,  and  performance  objectives  while  in 
compliance  with  the  statutory  requirements  and  agency  regulations 
ensuring  performance  of  all  necessary  actions  for  effective 
contracting,  ensuring  compliance  with  the  terms  of  the  contract,  and 
safeguarding  the  interests  of  the  United  States  Government  in  its 
contractual  relationships.  (Neuman,  2013b,  p.  34). 

Basically,  the  KO  works  to  support  the  PM  to  meet  the  program’s  goals,  but  the 
KO  must  adhere  to  all  laws,  regulations,  and  agency  policies. 

Contracting  officers  have  specific  responsibilities  defined  by  regulations. 
According  to  FAR  1.602-2,  a  contracting  officer  shall; 
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(a)  Ensure  that  the  requirements  of  1. 602-1  (b)  have  been  met, 
and  that  sufficient  funds  are  available  for  obligation; 

(b)  Ensure  that  contractors  receive  impartial,  fair,  and  equitable 
treatment; 

(c)  Request  and  consider  the  advice  of  specialists  in  audit,  law, 
engineering,  information  security,  transportation,  and  other 
fields  as  appropriate; 

(d)  Designate  and  authorize,  in  writing  and  in  accordance  with 
agency  procedures,  a  contracting  officer’s  representative 
(COR)  on  all  contracts  and  orders  other  than  those  that  are 
firm-fixed  price,  and  for  firm-fixed-price  contracts  and  orders 
as  appropriate,  unless  the  contracting  officer  retains  and 
executes  the  COR  duties,  (from  FAR,  2014,  Part  1.602-2) 

Army  contracting  officials  can  either  be  civilian  employees  authorized  to 
serve  in  1102  positions  (Office  of  Personnel  Management,  1983),  or  military 
personnel  serving  in  Functional  Area  97A,  Contracting  and  Industrial 
Management  Officer  (Jones,  2007),  or  Military  Occupational  Specialty  51 C, 
Acquisition,  Logistics,  and  Technology  Contracting  Non-Commissioned  Officer 
(U.S.  Army,  2011). 

In  the  past  decade.  Army  acquisition  officials  have  increasingly  operated 
in  joint  environments.  This  has  required  Army  contracting  personnel  to  interact 
with  contracting  specialists  from  the  other  services  and  other  governmental 
agencies.  While  all  DOD  personnel  operate  under  the  same  FAR  and  Defense 
Federal  Acquisition  Regulation  Supplement  (DFA'RS),  each  DOD  organization 
has  its  own  supplement  to  the  FAR,  and  each  organization  outside  the  DOD  may 
operate  under  different  guidance  promulgated  by  its  parent  organization  (e.g.. 
Army  FAR  Supplement  [AFARS],  Air  Force  FAR  Supplement  [AFFARS],  Navy 
Marine  Corps  Acquisition  Regulation  Supplement  [NMCARS],  Department  of 
State  Acquisition  Regulation  [DOSAR],  and  Department  of  Veterans  Affairs 
Acquisition  Regulation  [VAAR];  FAR,  2014,  Part  1.301). 

Contracting  Officer  Representative:  The  contracting  officer 
representative  (COR)  is  not  a  contracting  officer,  nor  does  he/she  hold  any 
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contracting  authority.  The  FAR  specifically  states  that  a  COR  “has  no  authority 
to  make  any  commitments  or  changes  that  affect  price,  quality,  quantity,  delivery, 
or  other  terms  and  conditions  of  the  contract  nor  in  any  way  direct  the  contractor 
or  its  subcontractors  to  operate  in  conflict  with  the  contract  terms  and  conditions” 
(from  FAR,  2014,  Part  1.602-2[d][5]).  The  function  of  a  COR  is  to  be  the  eyes 
and  ears  of  the  KO.  Ideally,  the  COR  would  be  a  technical  expert  and  assist  with 
monitoring  contractor  performance  on  a  specific  contract.  In  practice,  it  is  more 
likely  that  a  COR  will  not  have  a  high  degree  of  technical  expertise  in  the  field  of 
monitoring  contractor  performance. 

Functional  Area  Expert:  Functional  area  experts  have  a  specific  set  of 
skills  and  provide  the  acquisition  team  with  a  wide  range  of  talent  that  they  can 
tailor  to  each  service  contract.  They  provide  an  essential  service  by  reducing  the 
workload  and  technical  knowledge  required  by  the  contracting  officer.  Functional 
area  experts  include,  but  are  not  limited  to,  financial  managers  who  keep  the 
project  on  budget,  price/cost  estimators  who  analyze  the  price/cost  of  a  project, 
legal  officers,  engineers  who  have  technical  knowledge,  auditors,  small 
business/competition  advocates,  quality  assurance  representatives,  information 
technology  advisors,  and  logisticians  (Beers,  2011).  Service  contracts  can  range 
from  a  simple  janitorial  contract  to  highly  complex  multi-year  helicopter 
maintenance  and  repair  services  that  are  performed  on  multiple  bases  in  the 
continental  United  States;  overseas  bases  in  Europe,  Africa,  and  Korea;  and 
warzones  like  Afghanistan  and  Iraq. 

Step  2:  Review  Current  Strategy 

Step  2  begins  with  the  acquisition  team  creating  a  baseline  and  then 
analyzing  the  current  service  strategy  (Beers,  2011).  This  consists  of  looking  at 
historical  data,  identifying  problems,  projecting  any  type  of  modification  to  the 
mission,  and  getting  the  customer/end  user  to  define  the  key  performance 
outcomes  that  they  expect  from  this  contract.  This  is  by  far  the  most  important 
part  of  Step  2.  The  customer  must  be  able  to  define  what  they  expect  this 
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contract  to  accomplish  as  there  can  be  differences  between  what  is  said  and 
what  is  understood. 

For  example,  a  customer  requested  a  janitorial  cleaning  services  contract 
to  cover  portable  toilets  at  a  specific  location.  The  contracting  officer 
understands  that  this  contract  is  to  cover  cleaning  the  portable  toilets  and 
removing  waste,  but  goes  back  to  the  customer  to  define  key  requirements  to 
make  sure  everyone  has  the  same  view  of  the  performance  requirements.  The 
customer  states  that  they  want  the  vendor  to  check  the  level  of  toilet  paper  in 
each  portable  toilet  and  resupply  each  unit  as  necessary.  The  customer  also 
requires  that  the  services  contract  cover  the  removing  of  waste  water  and  trash 
from  the  hand  washing  station  next  to  the  portable  toilets  and  the  resupplying  of 
the  hand  sanitizer  solution  and  paper  towels  at  that  station.  What  the  customer 
originally  stated  as  the  requirement  and  what  he/she  thought  he/she  was  getting 
was  different  from  what  the  contracting  officer  understood  from  the  original 
request.  This  is  why  it  is  important  for  the  customer  and  acquisition  team  to 
define  the  requirements  and  expected  performance  outcomes. 

Step  3:  Market  Research 

As  Beers  (2011)  wrote,  “The  team  analyzes  the  marketplace  to  assess 
current  technology  and  business  practices,  competition  and  small  business 
opportunities,  and  existing  and  potential  new  sources  of  providing  the  service; 
the  team  then  determines  if  commercial  buying  practices  can  be  used”  (from 
Beers,  2011,  p.  56).  Ideally,  the  customer  or  end-user  would  have  conducted 
market  research  prior  to  giving  the  requirements  to  the  contracting  command. 
Market  research  can  provide  the  customer  and  the  contracting  officer  with  ideas 
about  the  types  of  services  required,  possible  vendors  to  provide  those  services, 
estimated  pricing  and  cost  data,  and  help  defining  what  the  customer  really 
needs.  One  of  the  main  reasons  for  doing  market  research  is  to  determine 
whether  a  service  is  commercially  available  or  is  a  government-unique  service. 
Commercial  services  tend  to  be  more  defined  and  have  more  vendors  to 
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compete  for  government  business.  Government-unique  services  can  be  more 
complicated  and  require  more  effort  to  define  the  requirement. 

Phase  2:  Develop 

The  second  phase  of  the  services  acquisition  process  is  the  Development 
phase.  This  phase  consists  of  Step  4,  Requirements  Definition,  and  Step  5, 
Acquisition  Strategy,  in  which  the  acquisition  team  develops  the  overall  plan  to 
procure  the  services.  The  requirements  and  performance  outcomes  that  the 
customer  provided  are  examined  and  refined  further.  Then  the  team  determines 
the  appropriate  path  to  contract  for  the  required  services. 

Step  4:  Requirements  Definition 

This  step  allows  the  acquisition  team  to  define  the  performance  objectives, 
determine  an  acceptable  variation,  and  choose  the  method  of  measuring 
performance  (Beers,  2011).  This  process  leads  the  team  to  develop  the 
Performance  Work  Statement  (PWS).  The  PWS  should  tie  directly  to  the 
requirements  and  is  used  to  define  how  the  vendor  will  be  evaluated  with  regard 
to  CPARS  objective  scores. 

The  following  is  an  example  of  defining  requirements.  The  Army  Mission 
and  Installation  Contracting  Command  (AM ICC)  received  a  request  from  a 
customer  to  purchase  aircraft  maintenance  services.  This  is  poorly  defined 
because  there  are  many  different  types  of  aircraft  that  require  different  types  of 
maintenance.  This  could  mean  ordering  spare  parts,  engine  repair,  repair  of 
proprietary  hardware,  repair  of  electrical  or  electronic  systems,  or  some  other 
type  of  maintenance.  The  requirement  does  not  say  where  the  maintenance 
services  will  be  performed.  When  does  the  customer  need  the  services?  How 
many  aircraft  will  be  serviced  or  how  much  maintenance  service  does  the 
customer  require?  The  requirement  provided  to  the  contracting  official  is  very 
vague,  which  does  not  allow  him/her  to  determine  what  the  customer  wants  or 
needs.  The  contracting  official  will  need  to  discuss  the  requirements  with  the 
customer  to  define  the  requirements. 


17 


step  5:  Acquisition  Strategy 

This  step  is  designed  to  select  the  contract  type  that  is  the  most  beneficial 
to  the  government  to  mitigate  risk,  provide  for  the  proper  amount  of  incentive  to 
the  vendor  to  complete  the  contract  objectives,  and  select  between  lowest  price 
and  tradeoffs  where  the  best  value  may  be  a  higher  priced  solution  (Beers, 
2011).  The  requirements  will  tell  the  vendor  what  the  performance  metrics  are, 
not  the  specific  methods  for  how  the  vendor  will  complete  the  job.  The  customer 
and  acquisition  team  care  only  about  the  performance  outcomes,  not  how  the 
vendor  accomplishes  the  task.  This  allows  the  vendor  to  be  innovative  about 
how  they  design  a  solution. 

Phase  3:  Execute 

The  third  phase  of  the  services  acquisition  plan  is  the  Execution  phase. 
This  phase  consists  of  Step  6,  Execute  Strategy,  and  Step  7,  Performance 
Management,  which  pulls  together  everything  that  the  customer  and  acquisition 
team  have  developed  up  to  this  point  and  culminates  in  the  requirement  going 
out  to  industry  in  the  form  of  a  request  for  proposals  (RFP).  The  final  part  of 
Phase  3  is  concerned  with  contract  performance  and  administration  processes 
designed  to  close  out  or  terminate  a  contract. 

Step  6:  Execute  Strategy 

This  step  involves  the  creation  of  synopsis  and  solicitation  documents  that 
“formally  communicates  to  industry  the  customer’s  requirements  and  business 
plan”  (from  Beers,  2011,  p.  57).  The  acquisition  team  may  release  a  draft  RFP  to 
solicit  industry  feedback  on  the  feasibility  of  the  requirements  and  to  get  a  feel  for 
industry  interest  in  performing  this  contract.  Once  the  requirement  is  ready  for 
release  to  the  public,  vendors  will  review  the  documents  and  create  a  solution  to 
meet  those  objectives.  The  acquisition  team  may  establish  a  competitive  range 
to  limit  the  number  of  applicants  to  an  economical,  efficient,  and  manageable 
level  as  their  time  and  resources  are  limited  and  cannot  process  100,  50,  or  even 
10  proposals  if  the  requirements  are  complex. 
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The  acquisition  team  will  evaluate  each  proposal  against  the  requirements 
specified  in  the  RFP  and  the  standards  and  performance  measures  included  in 
the  Performance  Work  Statement  (PWS).  The  team  will  not  evaluate  proposals 
against  each  other.  The  acquisition  team  will  recommend  to  the  source  selection 
authority  (SSA)  the  proposal  that  best  meets  the  RFP  requirements.  The  final 
decision  on  selecting  the  best  proposal  rests  with  the  SSA.  The  SSA  may  be 
one  or  more  levels  above  the  contracting  officer.  The  contracting  officer  will  then 
inform  all  the  non-selects  of  the  results.  If  there  is  some  discrepancy  with  the 
contract  award,  the  non-selects  will  have  10  days  from  the  contract  award  to 
protest  the  award.  At  this  point,  the  acquisition  team  will  move  into  the 
administrative  or  management  step. 

Step  7:  Performance  Management 

As  Beers  (2011)  observed,  this  step  “involves  two  key  areas: 
administering  contract  requirements,  such  as  invoicing  and  payments,  and 
managing  the  relationships  and  expectations  of  both  the  contractor  and 
customers  in  meeting  the  terms  of  the  contract  and  achieving  the  required 
mission  performance  results”  (from  Beers,  2011,  p.  57).  Many  in  the  contracting 
world  argue  that  this  step  is  where  the  real  work  begins. 

Contractor  performance  on  the  contract  must  be  monitored,  which  is 
where  the  COR  aids  the  contracting  officer.  The  COR  will  communicate  with  the 
contractor  and  contracting  officer  regularly  to  provide  progress  reports  and  help 
solve  problems  as  they  arise.  Contracts  can  be  terminated  for  default  (T4D), 
terminated  for  convenience  (T4C)  of  the  government,  or  closed  out  upon 
successful  completion.  While  contracts  should  be  closed  out  within  six  months, 
there  are  situations  in  which  contracts  remain  open  for  decades  while  the 
government  and  contractor  disagree  over  payments,  terms  and  conditions,  and 
settlement  of  any  court  appeals.  An  example  is  Boeing  determining  what  the 
final  labor  rate,  overhead  rate,  and  time  and  material  rates  are  for  a  cost 
reimbursement  type  contract.  Boeing  must  wait  until  all  other  product  lines  have 

provided  data  so  the  company  can  break  down  those  costs  to  the  relevant 
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contract  and  develop  these  rates  for  reimbursement.  The  government  will  then 
review  those  costs  for  accuracy  which  can  take  up  to  a  year  or  more  on  complex 
contracts.  If  there  is  a  discrepancy,  it  can  go  to  the  contract  agency  head,  the 
GAO,  or  the  Armed  Services  Board  of  Contract  Appeals,  which  can  add  years  to 
the  case.  The  contract  cannot  be  closed  out  until  every  last  discrepancy  is 
resolved,  even  if  it’s  as  simple  as  a  $0.01  difference  between  the  contracted 
price  and  the  contractor  invoice. 

Contractor  performance  must  be  monitored  regularly  to  comply  with 
contract  administration  requirements.  Contract  administration  can  cover  short 
periods  of  time  or  govern  multiple  periods,  depending  on  the  type  of  contract.  A 
contract  with  a  short  period  of  performance  may  see  the  source  selection, 
contract  award,  contract  performance,  and  closed  out  periods  all  occur  within  the 
same  fiscal  year.  In  this  instance,  a  contracting  officer  would  provide  the  CPARS 
with  an  evaluation  of  that  contractor’s  performance  on  that  particular  contract 
based  upon  the  evaluation  criteria  set  forth  in  the  contract.  A  contract  that 
crosses  fiscal  years,  has  multiple  option  years  of  the  same  contract,  or  is  a  multi¬ 
year  contract  (e.g.,  ship  building,  military  construction),  may  require  a  CPARS 
evaluation  report  on  the  contractor’s  performance  on  an  annual  or  more  regularly 
scheduled  basis  and  a  final  CPARS  evaluation  when  the  contract  closes  out. 
Military  personnel  move  on  a  regular  basis  so  the  PM,  KO,  or  COR  who  starts  on 
a  contract  source  selection  may  be  different  from  the  PM,  KO,  or  COR  who 
administers  the  contract,  who  might  differ  from  the  PM,  KO,  or  COR  who  closes 
out  the  contract.  Diligent  monitoring  of  contractor  performance  is  the  only  way  to 
accurately  evaluate  and  rate  a  contractor  in  CPARS  over  the  life  of  the  contract. 

Accurate  CPARS  report  cards,  objective  scores,  and  descriptive  narratives 
are  important  because  they  are  used  during  the  government’s  source  selection 
process.  As  described  in  Figure  1,  the  Services  Acquisition  Plan,  the 
government  will  execute  the  acquisition  strategy  by  selecting  a  contractor  and 
monitoring  performance.  Contractor  past  performance  can  play  a  significant  role 
in  determining  eligibility  and  competitiveness  in  the  source  selection  process. 
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Contractor  past  performance  can  demonstrate,  but  is  not  limited  to,  contractor 
responsiveness  to  customer  needs,  reliability,  contractor’s  past  performance  on 
relevant  contracts,  government/contractor  business  relationships,  disqualifying 
conditions  (debarment,  suspension,  termination  for  default,  failure  to  meet 
standards),  contractor/subcontractor  business  relationships,  and  factors  to 
mitigate  potential  risk.  Given  the  role  that  past  performance  information  plays  in 
the  source  selection  process,  it  is  important  to  review  the  CPARS  system.  The 
next  section  provides  an  overview  of  CPARS,  the  narrative  and  issues 
surrounding  CPARS  information. 

C.  CONTRACTOR  PERFORMANCE  ASSESSMENT  REPORTING 
SYSTEM 

The  CPARS  is  used  by  all  federal  agencies  to  evaluate  contractor 
performance.  The  purpose  behind  this  system  is  to  give  federal  procurement 
personnel  a  way  to  assess  “a  contractor’s  performance,  both  positive  and 
negative,  and  provide  a  record  on  a  given  contract  during  a  specified  period  of 
time”  (from  Hart  et  al.,  2013,  p.  20).  CPARS  report  cards  are  segmented  into  five 
distinct  business  sectors:  Systems,  Services,  Operations  Support,  Fuels,  and 
Information  Technology  (DON,  1997).  The  key  business  sector  “Services”  can 
be  subdivided  into  Professional/Technical  &  Management  Support  Services, 
Repair  &  Overhaul,  and  Installation  Services.  The  excerpt  below  is  from  the 
CPARS  manual  and  describes  the  types  of  contracts  that  are  considered 
“Service”  contracts. 

Services:  Generally,  all  contracted  services  except  those  related  to 
“Science  &  Technology,”  “Construction  &  Architect — Engineering 
Services,”  and  “Health  Care.” 

Professional/Technical  &  Management  Support  Services: 

Includes  all  consultant  services — those  related  to  scientific  and 
technical  matters  (e.g.,  engineering,  computer  software  engineering 
and  development),  as  well  as  those  related  to  organizational 
structure,  human  relations,  etc.  Includes  office  administrative 
support  services  (e.g.,  any  basic  or  applied  research  that  will  result 
in  new  or  original  works,  concepts,  or  applications,  but  does  not 
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include  contract  advice  on  the  feasibility  of  such  research,  as  well 
as  evaluation  of  research  results. 

Repair  &  Overhaul:  Services  related  to  the  physical  repair  and 
overhaul  of  aircraft  ground  vehicles,  etc.,  and  any  associated 
subsystems  or  components.  Includes  condition  evaluations  of 
individual  items  received  for  repair  or  overhaul,  but  does  not  include 
evaluations  of  the  feasibility  or  the  benefits  of  the  overall  project. 

Does  not  include  ship  repair  and  overhaul  which  is  included  under 
the  Systems  sub-sector  on  Shipbuilding. 

Installation  Services:  Includes  services  for  grounds  maintenance 
(grass  cutting,  shrubbery  maintenance  or  replacement,  etc.). 
Includes  services  related  to  cleaning,  painting,  and  making  minor 
repairs  to  buildings  and  utilities  services,  etc.  Includes  contracted 
security  and  guard  services.  Includes  installation  and  maintenance 
of  fencing.  Includes  minor  electrical  repairs  (e.g.,  replacing  outlets, 
changing  light  bulbs,  etc.).  Includes  minor  road  surface  repairs 
(patching  cracks,  filling  in  potholes,  etc.).  Includes  relocation  of 
individual  telephone  lines  and  connections.  Includes  snow 
removal.  (But,  see  also  “Construction  &  Architect/Engineering 
Services”  and  Information  Technology”  for  the  services  covered  by 
those  business  areas.),  (from  DON,  1997,  p.  A1-2) 

The  Services  Business  Sector  is  a  very  diverse  collection  of  services. 
Contracting  professionals  may  not  be  familiar  or  experienced  with  such  a  wide 
array  of  contracted  services  so  it  is  important  for  CORs,  KOs,  PMs,  and  the 
contract  management  team  to  work  together  to  ensure  that  the  contractor’s 
performance  is  clearly  and  accurately  recorded  in  the  CPARS  report  card. 

Past  performance  information  entered  into  CPARS  must  be  accurate  and 
timely  as  it  is  primarily  used  by  source  selection  officials  to  award  or  not  award  a 
contract  based  on  a  contractor’s  strengths  and  weaknesses  during  Step  6,  the 
Execute  Strategy  phase  of  the  Services  Acquisition  Plan.  As  explained  in  the 
CPARS  manual,  “The  value  of  CPARS  to  a  future  source  selection  team  is 
inextricably  linked  to  the  care  the  program  manager  takes  in  preparing  a  quality 
narrative  to  accompany  the  CPAR  scores”  (from  DON,  1997,  p.  1). 

The  CPARS  evaluation  assesses  at  least  five  distinct  areas  of  contractor 
performance  supported  by  objective  data  presented  in  Block  20  (DON,  1997). 
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These  areas  are  Quality  of  Product/Service,  Schedule,  Cost  Control,  Business 
Relations,  and  Management  of  Key  Personnel.  Objective  scores  are  used  by  the 
acquisition  team  to  rate  contractor  performance  in  these  five  areas.  There  are 
five  possible  scores  for  each  of  these  five  areas.  The  five  possible  scores  are 
unsatisfactory,  marginal,  satisfactory,  very  good,  and  exceptional  (also  shown  in 
Table  3). 

CPARS  includes  a  variance  assessment  that  assesses  the  variance 
between  current  costs  and  schedules  and  government  estimates.  Scoring  for 
Block  18  and  19  must  be  in  accordance  with  Appendix  B,  SPECIFIC  EXAMPLES 
OF  NARRATIVE  STATEMENTS  TO  AVOID  FROM  THE  CPAR  QUALITY 
CHECKLIST  (N.D.)  and  Evaluation  Ratings  Definitions  (DOD,  2011,  p  A2-1),  and 
is  described  in  detail  in  Chapter  III — Research  Methodology. 

Finally,  CPARS  includes  a  program  manager  narrative  that  is  a  statement 
of  the  facts  regarding  the  contractor’s  performance  on  a  particular  contract.  The 
program  manager  is  responsible  for  generation  and  content  of  the  narrative  as 
the  PM  must  sign  and  date  this  form  prior  to  sending  it  to  the  KO  for  review 
(DON,  1997).  The  narrative  comments  in  Block  20  should  directly  trace  back  to 
the  corresponding  objective  scores  in  Blocks  18  and  19  of  the  CPARS  report 
card  (DON,  1997). 

Consistent  and  objective  evaluations  of  contractor  performance  are 
essential,  which  is  why  a  series  of  checks  and  balances  were  designed  within  the 
CPARS  process  (Hart  et  al.,  2013).  As  stated  in  the  CPASRS  manual,  “Each 
assessment  must  be  based  on  objective  facts  and  be  supportable  by  program 
and  contract  management  data,  such  as  cost  performance  reports,  customer 
comments,  quality  reviews,  technical  interchange  meetings,  financial  solvency 
assessments,  and  functional  performance  evaluations”  (from  DON,  1997,  p.  2). 

Before  the  performance  assessment  can  be  finalized,  it  must  be  forwarded 
to  the  contractor  for  review  and  comment.  This  ensures  the  past  performance 
information  report  card  is  finalized  only  after  the  contractor  has  the  opportunity  to 
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provide  feedback  to  the  assessing  official.  The  contractor  is  not  required  to 
respond  but  must  have  the  opportunity  to  provide  inputs  to  the  CPARS 
evaluation.  Only  after  this  has  happened  can  the  approving  official  finalize  the 
report.  The  approving  official  is  someone  in  a  position  at  least  one  level  above 
the  program  manager  to  help  ensure  impartiality  and  objectivity  (DON,  1997). 
Even  with  these  checks  and  balances  in  place,  the  Army  and  the  DOD  as  a 
whole  have  had  below-standards  records  for  CPARS  compliance.  Table  1 
shows  the  compliance  rates  for  the  Army  and  the  larger  DOD  with  a  49%  and 
56%  in  2011  but  a  much  improved  compliance  rate  of  71%  and  74%, 
respectively,  for  2013  (GAO,  2013,  p.  9). 


DOD  Compliance  Rate  as  of 


Component 

2011-0ct-03 

2012-SeD-28 

2013-ADr-01 

Air  Force 

82% 

82% 

80% 

Navy 

66% 

69% 

72% 

Army 

49% 

60% 

73% 

Other  DOD 

32% 

61% 

71% 

Total  DOD 

56% 

66% 

74% 

Table  1 .  Percentage  of  Required  Assessments  Submitted  to  the 
Past  Performance  Information  Retrieval  System  (PRIRS) 
(from  GAO,  2013,  p.  9) 


The  following  section  addresses  several  issues  identified  by  the  GAO  and 
the  Office  of  Federal  Procurement  Policy  (OFPP)  in  the  area  of  documenting 
contractor  past  performance  information  in  CPARS  and  their  steps  to  correct  and 
improve  the  process. 

D.  INCREASED  SCRUTINY  OF  FEDERAL  CONTRACTING  AGENCIES 

An  increased  focus  on  contractor  past  performance  began  in  1994  with 
the  Federal  Acquisition  Streamlining  Act  (FASA),  in  which  Congress  directed  that 
government  contracting  agencies  should  use  contractor  past  performance 
information  as  one  of  the  award  decision  evaluation  factors.  As  a  result  of  the 
FASA,  the  FAR  was  updated  to  reflect  this  new  requirement,  and  the  OFPP 
began  releasing  guides  and  memoranda  for  the  purpose  of  instructing  federal 
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contracting  professionals  in  the  best  practices  for  collecting,  submitting,  and 
utilizing  contractor  past  performance  information. 

One  of  the  biggest  obstacles  in  the  way  of  achieving  more  reliably 
widespread  and  frequent  use  of  contractor  past  performance  information  in  the 
contracting  process  was  the  lack  of  a  centralized  reporting  system.  In  2002,  the 
DOD  directed  that  the  PPIRS,  a  web-enabled  contractor  past  performance 
information  database  and  retrieval  tool,  was  its  single  authorized  retrieval  system 
for  past  performance  information.  However,  a  GAO  report  from  2009  revealed 
that  “PPIRS  data  for  fiscal  years  2006  and  2007  indicates  that  only  a  small 
percentage  of  contracts  had  a  documented  performance  assessment”  (from 
GAO,  2009,  Executive  Summary).  Table  2  shows  the  disparity  between  the  total 
numbers  of  contracts  requiring  assessments  versus  contracts  with  an  actual 
completed  assessment.  In  2007,  the  Air  Force  led  the  way  with  47%.  In  2009, 
the  FAR  began  requiring  all  federal  agencies  to  submit  all  of  their  contractor 
performance  evaluations  to  PPIRS. 


Estimated  Contracts  Requiring  an  Assessment  That  Had  an  Assessment  In  Fiscal  Year  2007 

Estimated  contracts 

Contracts  with  an 

Department/agency 

requiring  an  assessment 

assessment 

Percent 

Air  Force 

2,795 

1,300 

47 

Navy 

3,879 

1,622 

42 

Army 

6,145 

1,971 

32 

Other  DOD 

1,408 

303 

22 

Homeland  Security 

4,131 

535 

13 

NASA 

3,706 

1,093 

29 

Energy 

840 

183 

22 

Total  percentage 

22.904 

7.007 

31 

Source:  GAO  analysis  of  data  from  DOD.  FPDS-NG.  and  PPIRS. 


Table  2.  DOD  CPARS  Requirement  versus  Actual  Reporting 
(from  GAO,  2009,  p.  12) 


In  2007,  the  GAO  released  a  report.  Use  of  Contractor  Performance 
Information,  detailing  its  study  of  how  contractor  past  performance  information 
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may  be  considered  in  the  contracting  process  and  what  issues  government 
contracting  agencies  have  encountered  with  the  use  of  this  information.  The 
report  identifies  many  of  the  complexities  that  go  into  the  process  of  evaluating 
contractor  past  performance  and  how  these  complexities  have  affected  GAO 
protest  decisions  (GAO,  2007). 

Contractor  past  performance  information  is  fundamentally  important  to  the 
contracting  process  from  the  beginning  during  the  pre-award  period  through  the 
end  of  contract  performance.  The  GAO  report  looked  at  several  ways  in  which 
government  contracting  finds  use  in  considering  contractor  past  performance. 
These  include  source  selection,  responsibility  determinations,  surveillance  of 
performance  under  the  current  contract,  and  suspension  and  debarment  (GAO, 
2007). 

A  GAO  report  from  2009  revealed  the  results  of  a  review  of  62 
government  contract  solicitations  from  fiscal  years  2007  and  2008  for  which 
contractors’  past  performance  information  was  considered  (GAO,  2009).  The 
researchers  also  interviewed  121  contracting  officials,  asking  them  about  how 
their  agencies  utilize  contractor  past  performance  information.  The  GAO  study 
showed  that  although  contractor  past  performance  information  was  being 
reviewed,  to  include  the  use  of  the  PPIRS  system,  that  “factors  other  than  past 
performance,  such  as  technical  approach  or  cost,  were  the  primary  factors  for 
contract  award  decisions”  (from  GAO,  2009,  Executive  Summary).  While  the 
FAR  does  allow  contracting  officials  broad  discretion  for  how  heavily  to  weight 
contractor  past  performance  relative  to  other  evaluation  factors  in  their  source 
selection  evaluation,  the  GAO  study  revealed  that  “a  majority  of  officials  told  us 
their  reluctance  to  rely  more  on  past  performance  was  due,  in  part,  to  their 
skepticism  about  the  reliability  of  the  information  and  difficulty  assessing 
relevance  to  specific  acquisitions”  (from  GAO,  2009,  p.  8). 

The  OFPP  followed  the  GAO  report  in  2009  with  a  memorandum  for  the 

chief  acquisition  officers  and  senior  procurement  executives  that  acknowledged 

the  “fragmented  methods”  that  agencies  used  to  collect  and  maintain  contractor 
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performance  information.  The  OFPP  stated  that  agencies  “maintain  evaluations 
in  internal  data  systems  that  are  not  available  to  acquisition  officers  outside  that 
agency”  (from  OFPP,  2009,  p.  1).  This  was  occurring  despite  the  FAR 
requirement  that  contractor  performance  information  be  shared  between 
agencies.  The  OFPP  issued  new  requirements  to  agencies  pursuant  to  FAR 
subpart  42.15  changes  effective  July  2009  mandating  submission  of  contractor 
performance  records  to  PPIRS.  The  OFPP  encouraged  government  contracting 
agencies  to  designate  specific  individuals  to  the  task  of  ensuring  that  “accurate, 
complete,  and  timely  information  is  submitted  to  PPIRS”  (from  OFPP,  2009,  p.  2). 

In  2011,  the  OFPP  issued  a  memorandum  to  share  the  findings  of  its 
review  of  agency  compliance  with  its  2009  directives  and  to  provide  additional 
recommendations  for  continued  improvement  in  the  collection  of  contractor  past 
performance  information.  Researchers  looked  at  nearly  700  performance  reports 
submitted  by  the  10  federal  contracting  agencies  that  make  up  the  bulk  of  the 
federal  contracting,  together  accounting  for  94%  of  federal  contract  obligations 
for  fiscal  year  2009  (OFPP,  2011).  This  review  took  place  during  the  period  in 
which  contracting  agencies  were  migrating  to  the  Contractor  CPARS  for  the 
purpose  of  submitting  their  contractor  performance  information  to  PPIRS. 

The  OFPP  found  that  in  2009,  the  DOD  had  conducted  required  past 
performance  evaluations  on  about  50%  of  contract  awards  and  that  the  quality  of 
those  submitted  were  of  varying  quality  with  most  deemed  lacking  in  sufficient 
information.  They  evaluated  the  contractor  performance  report  narratives  based 
on  how  well  they  addressed  the  four  required  rating  factors:  quality  of  the  product 
or  service,  ability  to  control  cost,  ability  to  meet  schedule,  and  quality  of  business 
relations  (OFPP,  2011).  The  DOD  has  used  CPARS  for  submitting  contractor 
past  performance  reports  since  2004  (GAO,  2013). 

The  OFPP  review  of  performance  reports  submitted  by  the  DOD  revealed 
the  following:  53.1%  contained  sufficient  narrative  for  quality  of  product/service, 
51%  contained  sufficient  narrative  for  schedule  control,  21.9%  contained 
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sufficient  narrative  for  cost  control,  and  50%  contained  sufficient  narrative  for 
business  relations  (OFPP,  2011). 

The  OFPP  issued  guidance  to  improve  the  quality  of  contractor  past 
performance  reporting.  Agencies  were  directed  to  establish  roles  and 
responsibilities  for  those  individuals  appointed  to  review  and  submit  performance 
reports  and  to  facilitate  proper  training  for  all  acquisitions  personnel.  A  new 
increased  focus  on  training  and  accountability  was  expected  to  significantly 
improve  contractor  performance  reporting.  Agencies  are  required  to  establish  an 
agency  point  of  contact  accountable  for  disseminating  guidance,  facilitating 
training,  developing  oversight,  and  identifying  improvements  to  the  submission 
process.  They  were  directed  to  institute  a  performance  report  review  process  for 
the  purpose  of  monitoring  their  quality  (OFPP,  201 1 ). 

In  2013,  the  OFPP  took  additional  steps  to  improve  the  quality  of 
contractor  performance  assessments.  In  March,  it  released  a  memorandum 
establishing  a  baseline  for  compliance  with  contractor  performance  reporting 
standards  and  setting  agency  performance  targets  for  fiscal  years  2013  through 
2015  (OFPP,  2013). 

A  2013  GAO  report  shows  that  measures  taken  by  the  OFPP  and  the  FAR 
Council  have  led  to  higher  rates  of  submission  for  performance  assessments 
within  the  DOD.  In  2007,  fewer  than  half  were  completed.  Submission  of 
required  assessments  increased  from  56%  in  October  2011  to  74%  by  April 
2013.  Flowever,  the  report  shows  that  the  DOD  is  still  failing  to  complete 
assessments  on  time.  As  Figure  2  shows,  a  large  portion  of  the  required 
assessments  are  completed  late  (i.e.,  after  the  120-day  requirement;  GAO, 
2013). 
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Figure  2. 
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Source  Contractor  Performance  Assessment  Reporting  System  (CPARS)  Program  Office 


DOD  CPARS  Reporting  Compliance  2010-2012  (from 
GAO,  2013,  p.  10) 


The  GAO  report  focuses  on  quantifying  the  effectiveness  of  the  measures 
taken  by  the  DOD  to  improve  the  quality  and  timeliness  of  its  contractor 
performance  reports  that  it  submitted  via  CPARS.  DOD  requirements  for 
contractor  performance  reports  (as  specified  in  the  CPARS  Guide)  specifies  that 
performance  assessments  must  be  completed  and  finalized  within  120  days  from 
the  end  of  the  contractor  evaluation  period  and  that  the  assessing  official  is 
responsible  for  preparing  and  finalizing  the  report.  The  report  indicates  that 
“although  the  CPARS  Guide  does  not  currently  specify  standards  for 
completeness  ...  a  recent  proposed  change  to  the  FAR  will  address 
completeness  by  providing  minimum  government-wide  standards  for  past 
performance  rating  elements.  Specifically,  the  proposed  rule  requires  that  all 
assessments  address,  at  a  minimum,  quality  of  product  or  service,  timeliness, 
and  management  or  business  relations”  (from  GAO,  2013,  p.  8).  As  of  June 
2013,  there  were  no  formally  established  standards  for  quality  and  completeness 
of  CPARS  performance  narratives. 

CPARS  data  is  not  and  should  not  be  the  only  source  of  information  used 
to  determine  contractor  past  performance  (DOD,  2012).  Other  sources  of  PPI 
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are  references  provided  by  said  contractor,  references  provided  by  other  federal 
contracting  personnel,  and  other  information  not  in  CPARS.  One  of  the 
limitations  of  CPARS  is  that  federal  contracting  personnel  are  not  required  to 
enter  PPI  into  CPARS  if  the  dollar  amount  of  the  specific  contract  does  not  meet 
the  minimum  reporting  threshold,  as  described  in  Figure  3.  While  the 
government  may  elect  to  enter  PPI  into  CPARS  below  these  thresholds,  this  is 
not  common  practice  and,  as  a  result,  leads  to  gaps  in  PPI  data  below  the 
$1  million  threshold  for  service  contracts. 


Business  Sector 

Dollar  Threshold 

Systems 

>  $5,000,000 

Ship  Repair  &  Overhaul 

>  $500,000 

Services 

>  $1,000,000 

Health  Care 

>  $150,000 

Operations  Support 

>  $5,000,000 

Fuels 

>  $150,000 

Information  Technology 

>  $1,000,000 

Figure  3.  DCD  Reporting  Thresholds 


E.  PREVIOUS  STUDIES 

This  is  the  seventh  study  of  a  series  of  research  projects  aimed  at 
services  contract  management.  The  original  research  began  in  2006  when  Apte, 
Ferrer,  Lewis,  and  Rendon  examined  a  growing  trend  of  DCD  acquisition 
workload  over  the  previous  decade.  As  noted  by  Hart,  Stover,  and  Wilhite 
(2013),  one  of  the  major  observations  of  this  research  was  a  66%  increase  in 
services  contracting  since  1999  but  no  corresponding  increase  in  contract 
management  personnel  (Apte  et  al.,  2006). 

The  second  research  project  was  an  exploration  of  supply  chain 

management  of  service  contracts  (Apte  &  Rendon,  2007).  In  general  terms,  the 

research  focused  upon  how  the  Air  Force  used  the  Air  Education  and  Training 

30 


Center  (AETC)  and  Acquisition  Management  and  Integration  Center  (AMIC) 
models  (Hart  et  al.,  2013).  Apte  and  Rendon  observed  that  these  processes 
enabled  cradle-to-grave  handling  of  a  service  acquisition  but  allowed  for  a 
communication  failure  as  program  managers  are  not  on  site  (Hart  et  al.,  2013). 

The  third  research  project  differed  from  the  previous  ones  in  that  this  was 
based  on  a  survey  of  Air  Force  and  Navy  service  contracts  (Hart  et  al.,  2013). 
Apte,  Apte,  and  Rendon  focused  upon  program  management  issues  and 
approaches  of  service  contracting  (Apte  et  al.,  2008).  This  was  relevant  to 
research  conducted  by  Hart,  Stover,  and  Wilhite  but  is  not  directly  relevant  to  the 
narrative  evaluation  in  CPARS. 

The  fourth  research  project  focused  on  services  contract  management 
was  published  in  2009  and  expanded  upon  the  third  research  project  by  including 
the  Army  in  the  survey.  Again,  problems  were  noted  with  the  acquisition  team 
and  communication  issues  (Hart  et  al.,  2013).  The  major  takeaway  from  this 
research  is  that  most  acquisition  professionals  believed  that  “their  organizations 
[did  not  have]  sufficient  positions  and  that  the  available  positions  were  [not] 
adequately  filled”  (from  Hart  et  al.,  2013,  p.  29).  Like  the  previous  research 
projects,  this  improves  the  knowledge  base  for  service  contracts  but  is  not 
directly  related  to  the  narrative  evaluation  in  CPARS. 

The  fifth  research  project,  conducted  by  Apte,  Apte,  and  Rendon  in  2010, 
collected  data  from  the  previous  studies  and  analyzed  them  for  DOD 
management  techniques  for  services  acquisition  (Hart  et  al.,  2013).  They 
focused  upon  improving  the  communication  of  contracting  personnel  in  the  Army, 
Air  Force,  and  Navy.  Like  the  previous  research  projects,  this  improves  the 
knowledge  base  for  service  contracts  but  is  not  directly  related  to  the  narrative 
evaluation  in  CPARS. 

The  sixth  research  project  was  completed  in  2012  by  Apte,  Apte,  and 
Rendon.  This  project  focused  upon  the  drivers  of  management  practices  of 
successful  service  contracts  in  the  Army  (Hart  et  al.,  2013).  Hagan,  Spede,  and 
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Sutton  surveyed  168  key  personnel  to  determine  the  definition  of  a  successful 
service  contract  in  the  Navy  (Hart  et  al.,  2013).  The  outcome  of  this  research 
was  a  correlation  of  modifications,  protests,  and  communications  between  the 
acquisition  team  members.  Like  the  previous  research  projects,  this  improves 
the  knowledge  base  for  service  contracts  but  is  not  directly  related  to  the 
narrative  evaluation  in  CPARS. 

The  seventh  and  most  germane  research  project  was  completed  in  2013 
by  Hart,  Stover,  and  Wilhite.  The  group  explored  the  definition  of  a  successful 
service  contract  by  looking  at  the  CPARS  report  cards  stored  within  the  PPIRS. 
A  local  database  was  created  from  715  Army  service  contract  CPARS  report 
cards  for  the  purpose  of  evaluating  success  determined  by  six  CPARS  evaluation 
areas:  Quality  of  the  Product/Service,  Schedule,  Cost  Control,  Business 
Relations,  Management  of  Key  Personnel,  and  Utilization  of  Small  Business 
(Hartetal.,  2013). 

Contractor  evaluation  scores  for  the  six  evaluation  areas  were  assigned  a 
numerical  value  by  the  researchers  according  to  the  adjectival  rating  scale  in 
Table  3.  The  research  calculated  the  average  scores  to  find  any  correlation  to 
success  or  failure  for  the  six  evaluation  areas.  This  rating  system  and  the 
associated  database  is  the  foundation  for  the  research  that  is  to  be  examined  in 
Chapter  III  and  Chapter  IV  of  this  research  project. 


Rating 

Score 

Exceptional 

5 

Very  Good 

4 

Satisfactory 

3 

Marginal 

2 

Unsatisfactory 

1 

Table  3.  Area  of  Contract  Evaluation  Scores 
(from  Hart  et  al.,  2013,  p.  44) 


Of  the  715  Army  service  contracts  reviewed,  22  were  deemed  failures 

because  they  received  a  rating  of  unsatisfactory  or  marginal  for  one  or  more  of 
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the  six  evaluation  areas  previously  listed  (Hart  et  al.,  2013).  Table  4  shows  the 
breakdown  of  the  715  contracts  reviewed  by  Hart,  Stover,  and  Wilhite  (2013). 


Failures 

Success 

Total 

Failure  Rate 

Contracts 

22 

693 

715 

3.08% 

Table  4.  Total  Contract  Information  (from  Hart  et  al.,  2013,  p.  44) 


Hart,  Stover,  and  Wilhite  (2013)  found  that  service  contracts  had  an 
average  failure  rate  of  approximately  3%  (see  Table  4).  They  cited  several 
factors  that  led  to  contract  failure,  but  a  common  theme  among  them  was  a  lack 
of  proper  training  for  source  selection  evaluators  (SSEs).  SSEs  must  be  able  to 
properly  evaluate  each  contractor’s  proposal  according  to  the  requirements  in  the 
RFP  during  the  source  selection  process.  Past  performance  information,  to 
include  objective  ratings  and  descriptive  narratives,  plays  an  important  role  in  the 
SSE’s  evaluation  of  the  contractor’s  proposal.  It  is  this  past  performance 
information  that  is  the  focus  of  our  research  project. 

The  seven  previous  service  contract-related  research  projects  have 
provided  the  background  and  the  insight  to  pursue  this  research  project.  CPARS 
report  cards,  PPI,  and  the  database  of  715  Army  service  contracts  generated  by 
Hart,  Stover,  and  Wilhite  form  the  core  of  the  analysis  of  program  manager 
narratives  and  their  relation  to  the  objective  scores  in  CPARS. 

F.  SUMMARY 

This  chapter  presented  an  overview  of  the  DOD  services  acquisition 
process  and  the  literature  associated  with  said  process.  The  chapter  began  with 
a  presentation  of  the  service  contract  management  process  as  outlined  in  Beers’ 
2011  Services  Acquisition  Plan.  Next,  the  chapter  reviewed  the  CPARS  to 
provide  a  framework  for  the  database  which  is  the  basis  for  this  research  project. 
Finally,  the  chapter  concluded  with  a  review  of  previous  research  projects  related 
to  the  service  contracting  field  of  study. 
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The  next  chapter  discusses  the  research  questions  that  this  research 
attempts  to  answer,  the  methods  used  to  conduct  the  research,  the  raw  data  to 
be  analyzed,  and  the  type  of  statistical  analysis  used  to  understand  and  explain 
the  relationships  of  the  data. 
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III.  RESEARCH  METHODOLOGY 


A.  INTRODUCTION 

This  chapter  discusses  how  the  research  was  conducted  and  how  the 
authors  answered  the  research  questions.  The  chapter  begins  by  describing 
how  the  objective  scores  and  narratives  are  entered  into  CPARS  and  how  it 
relates  to  a  successful  contract.  Then  the  chapter  contains  an  in-depth 
description  of  how  that  data  is  analyzed,  including  a  description  of  the  statistical 
methodology  used  to  perform  the  analysis.  The  last  part  of  this  chapter  describes 
the  interviews  that  were  conducted,  along  with  the  interview  question 
development  methodology. 

This  research  examines  the  value  of  CPARS  report  card  narratives  for 
service  contracts  as  they  relate  to  their  associated  objective  scores.  The 
methodology  includes  data  analysis  (including  both  sentiment  and  statistical 
analysis)  and  interviews  with  government  agency  contracting  professionals. 

B.  OBJECTIVE  SCORES  AND  NARRATIVES 

The  first  phase  of  data  analysis  performed  was  a  sentiment  analysis  of 
715  Army  service  contract  CPARS  report  card  narratives.  The  CPAR  Quality 
Checklist  was  used  as  a  basis  for  developing  the  criteria  for  the  categories  and 
values  for  the  analysis  {CPAR  Quality  Checklist,  n.d.).  Data  from  715  CPARS 
report  cards  of  service  contracts  from  2012  were  made  available  as  the  basis  of 
this  continuing  research  (Hart,  Stover,  &  Wilhite,  2013).  Each  sample  provided  a 
series  of  objective  scores  in  a  spreadsheet  that  was  associated  with  each 
narrative. 

Table  5  provides  an  example  of  objective  scores  for  the  six  evaluation 
areas  used  to  rate  a  contractor’s  performance. 
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Quality  of 
Product  or 
Service 

Schedule 

Cost 

Control 

Business 

Relations 

Management 
of  Key 
Personnel 

Utilization 
of  Small 
Business 

Min 

Score 

Success 
or  Failure 

VERY 

VERY 

VERY 

VERY 

VERY 

VERY 

GOOD 

GOOD 

GOOD 

GOOD 

GOOD 

GOOD 

4 

SUCCESS 

(4) 

(4) 

(4) 

(4) 

(4) 

(4) 

Table  5.  Example  of  Objective  Scores 


Table  6  provides  examples  of  narratives  that  describe  in  the  appropriate 
amount  of  detail  the  performance  of  the  contractor  for  that  particular  objective 
score.  For  example,  an  objective  score  of  very  good  for  the  area  of  quality  of 
product  or  service  does  not  tell  the  full  story.  The  narrative  associated  with  an 
objective  score  of  very  good  is  meant  to  explain  how  the  contractor  maintained  a 
very  good  level  of  quality  via  a  quality  control  program  that  was  operated 
efficiently  and  effectively  throughout  the  life  of  the  contract. 


Quality  of  Product/Service:  This  contractor  has  done  a  very  good  job  overall.  This  particular 
task  order  is  for  the  management  Firm  Fixed  Priced  piece  in  support  of  a  major  customer. 
Overall,  quality  of  the  products  and  services  remains  high  and  the  contractor  quality  control 
program  runs  efficiently  and  effectively. 

Schedule:  Contractor  continues  to  provide  products/services  on  time.  Contractor  continues  to  be 
very  adaptive  to  unanticipated  work  requests. 

Cost  Control:  Contractor  has  historically  produced  accurate  invoices.  This  is  certified  annually 
under  the  Contractor  Manpower  Reporting  system. 

Business  Relations:  Contractor  has  very  good  Business  Relations.  The  government  has  never 
encountered  any  difficulty  in  working  with  this  contractor  to  resolve  any  issue.  Contractor  has 
three  (3)  subcontractors  assisting  in  the  performance  of  this  operation.  Contractor  provided  daily 
reports  on  all  equipment  under  their  control  for  maintenance.  The  contractor  was  effective  in 
programs  such  as  equal  opportunity,  employee  incentive,  energy  conservation,  safety,  security, 
and  upward  mobility. 

Management  of  Key  Personnel:  All  key  personnel  have  performed  very  well. 

Utilization  of  Small  Business:  Contractor  utilizes  three  small  businesses  as  subcontractors  on 
this  task  order. 

Overall  Comments:  Contractor  continues  to  satisfy  customers.  The  government  frequently 
receives  positive  comments  on  the  end  product  received  from  this  contractor.  Contractor  provides 
a  very  good  service  through  the  cost  plus  contract.  The  government  definitely  would  recommend 
awarding  future  contracts  to  this  contractor. 

Given  what  I  know  today  about  the  contractor's  ability  to  execute  what  he  promised  in  his 
proposal,  I  DEFINITELY  WOULD  award  to  him  today  given  that  I  had  a  choice. 

Table  6.  Example  of  Narratives  Associated  with  Objective 

Scores 
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To  analyze  the  narratives  in  the  samples  and  see  how  they  relate  to  the 
objective  scores,  a  series  of  criteria  had  to  be  developed.  The  CPAR  Quality 
Checklist  was  used  as  a  basis  for  developing  the  criteria  {CPAR  Quality 
Checklist,  n.d.).  Narratives  were  assigned  several  scores  relating  to  their  quality, 
robustness,  compliance  with  directions  in  the  CPAR  Quality  Checklist,  and  their 
value  and  content  compared  to  their  related  objective  scores  from  the  CPARS 
report  cards. 

First,  we  answered  a  series  of  yes  or  no  questions.  These  questions  are 
listed  as  follows. 

First,  does  the  narrative  address  all  performance  areas  assessed  as  an 
objective  score?  This  means  that  if  one  of  the  five  objective  scores  was  marked 
as  not  applicable,  there  would  not  need  to  be  something  written  in  the  narrative 
specifically  mentioning  that  area.  Qtherwise,  there  should  be  something  written 
in  the  narrative  to  address  the  score  given  for  each  area  in  the  objective  scores. 

Second,  is  the  narrative  based  on  objective  data?  In  answering  this 
question,  objective  data  is  defined  as  not  being  influenced  by  personal  feelings  or 
opinions  of  the  person  inputting  the  data,  but  only  considering  and  representing 
facts. 

Third,  is  the  narrative  free  of  statements  to  avoid,  as  defined  by  the  CPAR 
Quality  Checklist?  The  following  phrases  (or  those  closely  similar)  qualify  as 
narrative  statements  to  avoid:  “outside  the  scope  of  the  contract,”  “in  our 
opinion,”  “appeared,”  “we  believe,”  “it  is  our  hope,”  “we  are  not  happy,”  “we  did 
not  like,”  “we  think,”  “could  be,”  and  “we  hope.”  Appendix  B  provides  an 
expanded  list  of  narrative  statements  to  avoid. 

Fourth,  is  the  narrative  for  each  performance  area  of  a  robust  and 
comprehensive  nature,  as  described  in  the  three  examples  of  the  CPAR  Quality 
Checklist,  labeled  “Block  20  -  insufficient  narrative  vs.  a  better  way  to  write  this”? 
Appendix  C  provides  specific  examples  of  insufficient  narratives. 
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Fifth,  could  a  contracting  layman  understand  the  work  that  has  been 
performed?  This  will  not  require  an  opening  statement  giving  a  synopsis  of  the 
contract  in  the  narrative  because  that  would  be  part  of  a  separate  data  field 
defined  as  a  sample  contract  effort  description  within  CPARS  (Block  17)  outside 
the  narrative  portion  (Block  20).  Rather,  the  question  would  answer  whether 
there  was  an  excessive  use  of  jargon  or  acronyms  in  the  narrative,  to  the  point 
that  even  someone  familiar  with  common  terms  associated  with  contracting 
would  be  unable  to  understand  the  discussion  in  the  narrative. 

Once  the  yes  or  no  questions  have  been  scored,  the  following  two 
questions  would  be  answered  with  a  possible  score  of  1  to  5. 

First,  is  the  narrative  beneficial  to  a  user  of  the  information  above  and 
beyond  the  objective  scores?  To  earn  a  score  of  1,  the  narrative  provides  an 
unsatisfactory  amount  of  beneficial  data  to  the  user  above  and  beyond  what  could 
be  gleaned  from  looking  over  the  objective  scores  assigned  in  Block  18.  To  earn  a 
score  of  2,  the  narrative  provides  a  marginal  amount  of  beneficial  data  to  the  user 
above  and  beyond  what  could  be  gleaned  from  looking  over  the  objective  scores 
assigned  in  Block  18.  To  earn  a  score  of  3,  the  narrative  provides  a  satisfactory 
amount  of  beneficial  data  to  the  user  above  and  beyond  what  could  be  gleaned 
from  looking  over  the  objective  scores  assigned  in  Block  18.  To  earn  a  score  of  4, 
the  narrative  provides  a  very  good  amount  of  beneficial  data  to  the  user  above  and 
beyond  what  could  be  gleaned  from  looking  over  the  objective  scores  assigned  in 
Block  18.  To  earn  a  score  of  5,  the  narrative  provides  an  exceptional  amount  of 
beneficial  data  to  the  user  above  and  beyond  what  could  be  gleaned  from  looking 
over  the  objective  scores  assigned  in  Block  18. 

Second,  does  the  narrative  correlate  to  the  objective  scores  that  have 
been  assigned?  To  earn  a  score  of  1,  more  than  one  of  the  performance  areas 
described  in  the  narrative  is  contradictory  to  the  objective  scores  assigned  in 
Block  18.  To  earn  a  score  of  2,  one  (but  no  more  than  one)  of  the  performance 
areas  described  in  the  narrative  is  contradictory  to  the  objective  score  assigned 

in  Block  18.  To  earn  a  score  of  3,  the  narrative  is  merely  satisfactory  in 
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describing  accurately  why  the  objective  scores  are  assigned  as  they  are  in  Block 
18.  To  earn  a  score  of  4,  the  narrative  is  very  good  in  describing  accurately  why 
the  objective  scores  are  assigned  as  they  are  in  Block  18.  To  earn  a  score  of  5, 
the  narrative  is  exceptional  in  describing  accurately  why  the  objective  scores  are 
assigned  as  they  are  in  Block  18. 

The  next  step  in  conducting  our  analysis  of  the  narratives  was  to  figure  out 
the  best  way  to  go  about  handling  the  sheer  volume  of  715  report  cards  while  at 
the  same  time  thoroughly  reviewing  each  report  card.  Software  was  an  initial 
thought,  but  after  looking  at  this  option  in  depth,  it  became  obvious  that  the 
process  of  learning  how  to  use  and  manipulate  the  software  to  give  the  desired 
output  would  be  more  time  consuming  than  it  was  worth.  For  this  sample  size,  it 
turns  out  that  it  would  be  best  to  conduct  an  Inter-Rater  Reliability  Test  (Gwet, 
2008)  to  determine  a  baseline  agreement  amongst  the  raters  of  the  questions 
outlined  previously,  and  then  go  through  each  narrative  thoroughly  by  the  raters. 
If  this  could  prove  that  the  raters  were  trained  and  could  rate  each  sample 
similarly,  than  each  of  the  715  samples  would  not  need  to  be  analyzed  by  each 
researcher.  Using  this  approach,  the  715  samples  could  be  split  up  amongst  the 
raters  and  would  yield  similar  results  across  each  rater  for  the  same  sample. 

After  conducting  training  on  the  procedures  outlined  above  in  conducting 
each  score,  a  sample  of  30  reports  was  randomly  selected  from  the  original  715 
and  the  reports  were  independently  rated  by  each  rater.  Appendix  A  provides 
the  results  of  the  Inter-Rater  Reliability  Testing. 

To  score  each  set  of  scores  for  the  yes  or  no  questions,  a  score  of  3  was 
given  if  all  three  of  the  scores  matched.  The  only  other  possibility  for  the  yes  or 
no  questions  is  for  two  of  the  scores  to  match  and  the  other  to  not  match 
(highlighted  in  red).  In  this  case,  a  score  of  1  was  assigned  since  two  of  the 
scores  had  a  50%  rate  of  agreeing  to  the  other  scores  while  the  remaining  score 
had  a  0%  rate  of  agreeing  to  the  other  scores.  In  the  end,  the  scores  for  each 
set  of  scores  was  summed  and  then  divided  by  90  to  get  the  total  agreeability 
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percentage  in  each  category.  The  acceptable  threshold  was  set  at  whether  all 
raters  could  match  scores  for  each  sample  with  a  minimum  of  80%  reliability. 

The  results  for  the  yes  or  no  questions  show  the  percentage  of 
agreeability  between  each  rater’s  scores  of  the  30  randomly  selected  reports. 
There  was  100%  agreeability  on  whether  the  narrative  addresses  all 
performance  areas  assessed.  There  was  84.44%  agreeability  on  whether  the 
narrative  was  based  on  objective  data.  There  was  97.877%  agreeability  on 
whether  the  narrative  was  free  of  statements  to  avoid.  There  was  80% 
agreeability  on  whether  the  narrative  was  robust  and  comprehensive.  There  was 
91.11%  agreeability  on  whether  a  contracting  layman  could  understand  the  work 
performed. 

For  the  questions  that  were  assigned  scores  of  1  to  5,  the  scores  show  the 
percentage  agreeability  of  whether  the  raters  could  agree  in  their  results  while 
allowing  for  a  1  point  difference  amongst  raters  with  no  penalty.  The  threshold 
was  set  at  whether  all  raters  could  match  scores  for  each  sample  within  a  1  point 
difference  at  a  minimum  80%  reliability.  If  a  set  of  scores  all  matched,  a  score 
of  3  was  given  for  the  set.  If  a  set  of  scores  had  two  occurrences  of  scores  within 
1  (for  example,  1,  2,  3  or  3,  4,  5),  a  score  of  2  was  given  for  the  set.  If  a  set 
of  scores  had  only  one  occurrence  of  scores  within  1  (example  2,  4,  5),  a  score 
of  1  was  given  for  the  set.  If  there  had  been  an  instance  where  scores  of 
1,  3,  and  5  had  been  given,  a  score  of  0  would  have  been  given  for  the  set  (this 
did  not  occur). 

The  results  show  the  percentage  of  agreeability  between  each  of  the 
rater’s  scores  within  a  score  of  1  based  on  the  30  randomly  selected  samples. 
There  was  93.33%  agreeability  within  1  on  whether  the  narrative  is  beneficial 
above  and  beyond  the  objective  scores.  There  was  80%  agreeability  within  1  on 
whether  or  not  the  narrative  correlates  to  the  objective  scores  assigned. 

Because  all  of  the  total  agreeability  percentages  were  able  to  meet  the 
minimum  80%  threshold,  each  rater  went  through  -230  of  the  remaining  report 
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cards  independently  and  the  results  were  combined  and  analyzed  as  if  all  three 
raters  had  gone  through  all  715  report  cards.  The  next  section  discusses  the 
methodology  for  the  statistical  analysis  of  the  database. 

C.  METHODOLOGY  FOR  DATABASE  STATISTICAL  ANALYSIS 

An  independent  2-sample  test  for  Proportions  was  performed  to  find  the 
statistical  significance  of  the  proportions  of  the  outcomes  for  successful  versus 
unsuccessful  contracts  for  the  questions  that  have  only  two  possible  outcomes 
(yes  or  no).  The  p  value  in  the  hypothesis  test  shows  the  level  of  statistical 
significance.  A  p  value  of  less  than  .05  shows  statistical  significance.  A  p  value  of 
less  than  .01,  therefore,  shows  a  much  higher  degree  of  statistical  significance 
than  even  the  .05  values. 

A  chi-square  test  was  performed  to  show  collectively  whether  the  results 
for  the  questions  that  have  five  possible  outcomes  were  statistically  different  for 
successful  versus  unsuccessful  contracts.  The  p  value  used  in  the  chi-square 
test  shows  the  level  of  statistical  significance  and  the  results  are  interpreted  in 
the  same  way  as  in  the  hypothesis  test  that  compares  two  independent 
proportions  (explained  in  the  previous  paragraph).  The  next  section  discusses 
the  interview  questions  that  were  asked  government  contracting  personnel. 

D.  INTERVIEWS  OF  GOVERNMENT  CONTRACTING  PERSONNEL 

In  support  of  the  data  analysis,  interviews  with  contracting  professionals 
from  two  DOD  contracting  agencies  were  conducted.  These  interviews  consisted 
of  a  series  of  questions  asking  the  subjects  about  how  they  use  and  to  what 
extent  they  value  CPARS  and  other  sources  of  contractor  past  performance 
information  and  to  what  extent  they  value  the  narratives  compared  to  the 
performance  objective  scores. 

Our  research  includes  the  study  of  human  subjects  and  therefore,  as 
directed  by  regulation  from  the  Department  of  Health  and  Human  Services,  the 
research  interviews  required  approval  from  an  institutional  review  board  (IRB) 
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under  the  guidance  of  the  NPS  Office  for  Human  Research  Protections  (OHRP) 
specialist.  The  purpose  of  the  IRB  is  to  protect  human  subjects  in  research  and 
to  ensure  that  basic  ethical  principles  are  adhered  to. 

We  conducted  interviews  at  two  geographically  diverse  DOD  contracting 
agencies.  The  interviewees  had  some  combination  of  significant  experience 
submitting  contractor  past  performance  information  to  CPARS  and/or  retrieving 
from  the  Past  Performance  Information  Retrieval  System  (PPIRS)  past 
performance  information  for  use  in  contract  administration  or  source  selection. 
Interview  subjects  were  chosen  by  their  respective  department  heads. 

E.  INTERVIEWS  QUESTION  DEVELOPMENT  METHODOLOGY 

Interviews  with  contracting  professionals  who  are  experienced  with  the 
CPARS  and  PPIRS  systems  provide  the  requisite  context  and  depth  of 
knowledge  needed  to  accurately  and  robustly  analyze  and  interpret  the  statistical 
data  derived  from  the  CPARS  narratives.  The  specific  interview  questions  were 
designed  to  avoid,  to  the  extent  practicable,  injection  of  bias  by  steering 
respondents.  Interviewing  subjects  at  two  government  agencies  with  significantly 
different  missions  and  contract  award  types  facilitates  the  collecting  of 
information  across  a  wide  spectrum  of  knowledge  of  the  CPARS/PPIRS  systems. 
Interviewing  individuals  with  varying  roles  in  collecting,  submitting,  and  utilizing 
contractor  past  performance  information  provides  us  with  responses  across  a 
diverse  spectrum  of  experience.  A  complete  listing  and  explanation  of  the 
interview  questions  can  be  found  in  Appendix  D. 

Following  this  second  phase  of  research,  the  study  moved  into  analyzing 
CPARS  contractor  past  performance  report  card  narratives  and  comparing  that 
data  to  the  information  retrieved  from  these  contracting  professional  interviews. 

F.  SUMMARY 

This  chapter  presented  an  overview  of  the  methodology  for  collecting  the 
data,  analyzing  the  CPARS  narratives,  and  conducting  interviews  of  DOD  agency 
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contracting  professionals.  The  objective  scores  were  introduced  and  discussed. 
Next,  the  chapter  reviewed  the  CPAR  Quality  Checklist  to  provide  a  framework 
for  analyzing  the  database  which  is  the  basis  for  this  research  project.  The 
methodology  for  the  database  statistical  analysis  was  then  explained.  Finally, 
the  chapter  concluded  with  a  review  of  interview  questions  posed  to  DOD 
contracting  professionals.  The  next  chapter  discusses  the  results  of  the  statistical 
analysis,  responses  to  interview  questions  by  DOD  contracting  professionals, 
and  present  findings  that  answer  the  research  questions  posed  in  Chapter  III. 
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IV.  ANALYSIS 


A.  INTRODUCTION 

This  chapter  reveals  the  results  of  the  analysis  that  examined  the  value  of 
CPARS  report  card  narratives  for  service  contracts  as  they  relate  to  their 
associated  objective  scores.  This  chapter  begins  with  an  overview  of  the  data 
used  to  support  the  research  presented  in  this  report.  The  overview  first 
describes  the  database  created  for  this  report  in  order  to  show  from  where  the 
results  of  the  statistical  analysis  were  derived.  The  overview  then  provides  a 
description  of  the  series  of  interviews  that  were  conducted  to  show  from  where 
the  results  of  the  interview  findings  came.  Once  the  database  and  the  interviews 
are  described,  the  chapter  explains  the  statistical  analysis  findings  in  detail. 
Following  the  description  of  the  statistical  analysis  findings,  this  chapter 
describes  the  interview  findings  with  each  finding  supported  by  specific 
commentary  from  the  interviews. 

B.  OVERVIEW  OF  THE  DATA 

The  database  used  for  the  statistical  analysis  portion  of  this  project  is 
limited  to  the  715  Army  service  contracts  analyzed  by  a  previous  NFS  MBA 
project  (Hart  et  al.,  2013).  The  selection  criteria  for  those  715  contracts  was 
limited  to  Army  service  contracts,  “contracts  only  submitted  by  Mission 
Installation  Contracting  Commands  (MICCs),  and  contracts  from  only  five  MICC 
offices”  (from  Hart  et  al.,  2013).  Of  note,  out  of  the  715  contract  reports 
contained  in  the  database,  22  were  labeled  as  failures  which  “resulted  in  a  total 
contract  failure  rate  of  3.08%”  (from  Hart  et  al.,  2013,  p.  56).  The  determination 
for  which  contracts  should  be  specified  as  failures  “was  determined  by  whether  a 
contract  received  a  marginal  or  unsatisfactory  objective  rating  in  quality  of 
product/service,  schedule,  cost  control,  business  relations,  management  of  key 
personnel,  or  utilization  of  small  business”  (from  Hart  et  al.,  2013,  p.  50). 
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Interviews  with  contracting  professionals  from  two  DOD  contracting 
agencies  were  conducted  and  provided  the  basis  for  the  interview  analysis. 
These  interviews  consisted  of  a  series  of  questions  asking  the  subjects  about 
how  they  use  and  to  what  extent  they  value  CPARS  and  other  sources  of 
contractor  past  performance  information  and  to  what  extent  they  value  the 
narratives  compared  to  the  performance  objective  scores.  Appendix  D  provides 
the  specific  questions  asked  during  the  interviews  and  the  reasoning  behind 
those  questions. 

C.  STATISTICAL  ANALYSIS  FINDINGS 

Next,  we  reveal  the  findings  from  our  statistical  analysis.  First,  we  describe 
each  of  those  findings  in  detail.  We  cover  whether  the  narratives  address  all 
performance  areas  assessed,  whether  the  narratives  are  based  on  objective 
data,  whether  the  narratives  are  free  of  statements  to  avoid,  whether  the 
narratives  are  robust  and  comprehensive,  and  whether  the  narratives  are  written 
so  that  a  contracting  layman  should  understand  the  work  performed.  Then,  we 
cover  whether  the  narrative  provides  beneficial  data  above  and  beyond  what 
could  be  gleaned  from  looking  over  the  objective  scores  assigned.  Lastly,  we 
cover  how  well  the  narrative  sentiment  matches  up  with  the  objective  scores 
assigned.  At  the  end  of  this  section,  a  summary  chart  of  the  statistical  analysis 
findings  is  provided  in  Table  7. 

1.  Narrative  Addresses  All  Performance  Areas  Assessed? 

Overall,  the  narratives  addressed  all  performance  areas  assessed  82%  of 
the  time.  This  means  that  -18%  of  the  time,  contracting  professionals  are  not 
submitting  to  CPARS  contractor  performance  narratives  in  accordance  with  the 
guidelines  provided  in  the  CPARS  user’s  manual  for  this  requirement.  This  was 
less  problematic  with  unsuccessful  contracts  at  -95%  than  with  successful 
contracts  at  -81%.  The  difference  in  the  proportion  of  times  that  the  narrative 
addressed  all  performance  areas  assessed  in  successful  and  unsuccessful 
contracts  is  statistically  significant  (p  <  .05). 
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2.  Narrative  Is  Based  on  Objective  Data? 

Overall,  the  narratives  were  based  on  objective  data  -77%  of  the  time.  This 
means  that  -23%  of  the  time,  contracting  professionals  were  not  submitting  to 
CPARS  contractor  performance  narratives  in  accordance  with  the  guidelines 
provided  in  the  CPARS  user’s  manual  for  this  requirement.  However,  in 
unsuccessful  contracts,  the  narratives  were  based  on  objective  data  100%  of  the 
time.  This  is  significantly  different  from  the  -77%  in  successful  contracts  (p  <  .01 ). 

3.  Narrative  Is  Free  of  Statements  to  Avoid? 

Overall,  the  narratives  were  free  of  statements  to  avoid  -97%  of  the  time. 
This  means  that  -3%  of  the  time,  contracting  professionals  were  not  submitting 
to  CPARS  contractor  performance  narratives  in  accordance  with  the  guidelines 
provided  in  the  CPARS  user’s  manual  for  this  requirement.  This  was  slightly 
more  problematic  with  unsuccessful  contracts  at  -86%  than  with  successful 
contracts  at  -97%  (p  <  .01 ). 

4.  Narrative  Is  Robust  and  Comprehensive? 

Overall,  the  narratives  were  robust  and  comprehensive  -63%  of  the  time. 
This  means  that  -37%  of  the  time,  contracting  professionals  were  not  submitting 
to  CPARS  contractor  performance  narratives  in  accordance  with  the  guidelines 
provided  in  the  CPARS  user’s  manual  for  this  requirement.  This  was  less 
problematic  with  unsuccessful  contracts  at  -91%  than  with  successful  contracts 
at  -62%  (p<  .01). 

5.  Could  a  Layman  Understand  Work  Performed? 

Overall,  the  narratives  were  written  so  that  a  contracting  layman  should 
understand  the  work  performed  -64%  of  the  time.  This  means  that  -36%  of  the 
time,  contracting  professionals  were  not  submitting  to  CPARS  contractor 
performance  narratives  in  accordance  with  the  guidelines  provided  in  the  CPARS 
user’s  manual  for  this  requirement.  This  was  less  problematic  with  unsuccessful 
contracts  at  -82%  than  it  is  with  successful  contracts  at  -64%  (p  <  .05). 
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6. 


Is  the  Narrative  Beneficial  Above  and  Beyond  Objective 
Scores? 


As  previously  described  in  the  methodology  chapter,  the  chi-square  test 
determines  whether  distributions  of  scores  are  statistically  different.  Using  this 
test,  we  determined  that  there  was  a  difference  between  successful  and 
unsuccessful  contracts  in  whether  the  narratives  were  beneficial  above  and 
beyond  the  objective  scores.  Unsuccessful  contracts  tended  to  have  more 
beneficial  CPAR  report  card  narratives  than  successful  contracts. 

Overall,  the  narrative  provided  an  unsatisfactory  amount  of  beneficial  data 
to  the  user  -12%  of  the  time.  However,  there  were  no  unsuccessful  contracts 
that  provided  an  unsatisfactory  amount  of  beneficial  data.  The  narrative  provided 
a  marginal  amount  of  beneficial  data  -22%  of  the  time.  There  were  no 
unsuccessful  contracts  that  provided  a  marginal  amount  of  beneficial  data.  The 
narrative  provided  a  satisfactory  amount  of  beneficial  data  -28%  of  the  time.  The 
narrative  provided  a  very  good  amount  of  beneficial  data  -21%  of  the  time.  The 
narrative  provided  an  exceptional  amount  of  beneficial  data  -18%  of  the  time. 
This  was  much  more  likely  to  occur  with  unsuccessful  contracts  than  with 
successful  contracts  at  -17%. 

7.  Does  the  Narrative  Correiate  to  Objective  Scores  Assigned? 

We  again  used  the  chi-square  test  and  determined  that  there  was  not  a 
difference  between  successful  and  unsuccessful  contracts  in  whether  the 
narrative  correlates  to  the  objective  scores  assigned. 

Overall,  the  narrative  sentiment  was  contradictory  to  more  than  one  of  the 
objective  scores  assigned  -2%  of  the  time.  The  narrative  sentiment  was 
contradictory  to  one  of  the  objective  scores  assigned  -6%  of  the  time.  The 
narrative  sentiment  was  satisfactory  in  describing  accurately  why  the  objective 
scores  were  assigned  as  they  were  -28%  of  the  time.  The  narrative  sentiment 
was  very  successful  in  describing  accurately  why  the  objective  scores  were 
assigned  as  they  were  -40%  of  the  time.  The  narrative  sentiment  was 
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exceptionally  successful  in  describing  accurately  why  the  objective  scores  were 
assigned  as  they  were  -24%  of  the  time. 

Table  7  is  a  summary  chart  of  the  statistical  database  analysis  that  we 
have  just  described  in  detail. 


Result 

Overall 

Unsuccessful 

Contract 

Successful 

Contract 

Successful  vs 

Unsuccessful 

Statistically 

Significant? 

P  Value 

Narrative 

Addresses  All 

Performance 

Areas  Assessed? 

No 

18,46% 

4.55% 

18.90% 

Yes 

0.043 

Yes 

81.54% 

95.45% 

81.10% 

Namative  Based 
On  Objective 
Data? 

No 

22.52% 

0.00% 

23.23% 

Yes 

<.01 

Yes 

77.48% 

100.00% 

76.77% 

Nan-ative  Is  Free 

Of  Statements  To 

Avoid? 

No 

3.08% 

13.64% 

2.74% 

Yes 

<.01 

Yes 

96.92% 

86.36% 

97.26% 

Namative  Is 

Robust  & 
Comprehensive? 

No 

37.48% 

9.09% 

38.38% 

Yes 

<.01 

Yes 

62.52% 

90.91% 

61.62% 

Could  A  Layman 

Understand  Work 

Performed? 

No 

35.52% 

18.18% 

36.08% 

Yes 

0.042 

Yes 

64,48% 

81.82% 

63.92% 

Is  The  Narrative 

Beneficial  Above 
&  Beyond 
Objective  Scores? 

Score  1 

11.61% 

0.00% 

12.12% 

Yes 

<.01 

Score  2 

21.68% 

0.00% 

22.37% 

Score  3 

27.97% 

18.18% 

28.28% 

Score  4 

20.84% 

27.27% 

20.63% 

Score  5 

17.76% 

54.55% 

16.59% 

Does  The 

Narrative 

Correlate  To 
Objective  Scores 
Assigned? 

Score  1 

1.82% 

4.55% 

1.88% 

No 

0.141 

Score  2 

6.43% 

4.55% 

6.49% 

Score  3 

27.83% 

9.09% 

28.43% 

Score  4 

40.42% 

40.91% 

40.40% 

Score  5 

23.36% 

40.91% 

22.80% 

Table  7.  Results  of  Statistical  Database  Analysis 
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D.  INTERVIEW  FINDINGS 

As  previously  discussed,  we  interviewed  contracting  professionals  from 
two  geographically  diverse  DOD  contracting  agencies.  We  did  this  to  find  out 
how  they  use  and  to  what  extent  they  value  CPARS  and  other  sources  of 
contractor  past  performance  information.  We  also  wanted  to  find  out  to  what 
extent  they  value  the  narratives  compared  to  the  objective  scores.  In  the  process 
of  conducting  these  interviews,  we  made  some  interesting  discoveries  that  could 
be  useful  in  improving  the  contracting  process  specifically  related  to  the  narrative 
portion  of  past  performance  assessment  reports.  Below  are  the  eight  findings 
along  with  specific  excerpts  from  the  interviews  supporting  each  finding. 

1.  CPARS  Is  Still  Often  Not  Reliable,  Robust,  or  Comprehensive 
Enough 

The  information  in  CPARS/PPIRS  is  oftentimes  not  reliable,  robust,  or 
comprehensive  enough  to  allow  source  selection  officials  to  place  a  significant 
enough  weight  on  past  performance.  This  finding  was  also  noted  in  several  GAO 
and  OFPP  memorandums  mentioned  earlier  in  the  literature  review  chapter  of 
this  report. 

The  Past  Performance  Information  Retrieval  System  (PPIRS)  is  an 
information  system  that  draws  from  the  reports  entered  into  CPARS.  The 
contracting  professionals  we  interviewed  who  deal  regularly  with  utilizing  the  data 
in  PPIRS  felt  that  if  the  information  in  PPIRS  were  more  reliable  and  consistent, 
they  would  be  able  to  give  more  weight  to  it  when  making  source  selection 
decisions.  The  interviewees  stated  that  when  they  knew  there  should  be 
something  in  PPIRS  on  a  particular  contractor  for  a  particular  type  of  contract 
and  dollar  amount,  they  would  often  search  for  the  contractor’s  name  in  CPARS, 
only  to  receive  this  response:  “no  information  has  been  entered.” 

Interviewee  Number  2  made  the  recommendation  that  PPIRS  would  be  a 
more  useful  tool  if  source  selection  teams  could  get  more  information  in  there 
from  end  users.  Interviewee  Number  2  was  hesitant  because  this  recommended 
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improvement  will  likely  create  more  work  for  both  the  end  user  and  the  contract 
specialist,  but  suggested  that  DOD  may  want  to  lower  the  dollar  value  threshold 
to  help  contracting  offices  that  normally  deal  with  lower  value  contracts. 

Interviewee  Number  5  stated  that  the  CORs  and  contracting  officers  know 
generally  how  the  report  is  supposed  to  be  done,  “but  it  is  just  something  that  is 
not  executed  as  well  as  it  could  be.”  Interviewee  Number  5  stated  that 
information  from  CPARS/PPIRS  is  not  really  helpful  in  source  selection  and 
contract  administration  because  the  database  is  not  maintained  well  enough  and 
not  enough  thought  is  put  into  them.  Interviewee  Number  5  stated. 

Mostly  what  I  see  is  there  is  either  satisfactory  or  excellent  reviews 
even  if  there  are  problems,  the  problems  are  not  always 
documented  and  I  think  that  in  one  case  at  least  it  is  actually  the 
same  CPARS  pasted  in  to  each  year,  each  period  and  you  could 
tell  that  even  the  dates  were  actually  off  sometimes.  So  it  is  not 
something  that  we  put  enough  thought  into,  which  makes  it  not  a 
very  useful  tool  in  source  selections,  especially  when  you  know 
other  government  agencies  might  be  a  little  bit  similar  in  their 
practices  and  it  is  not  really  something  ...  and  it  is  something  that 
you  need  to  take  with  a  grain  of  salt.  I  think  when  we  are  going 
through  source  selection  really  what  we  look  for  is  the  offeror’s 
technical  understanding  of  the  requirements.  Sometimes  it  is 
obvious  that  there  is  a  little  bit  more  business  development 
language  than  actual  understanding  of  the  work  and  that  seems  to 
be  more  of  the  focus  than  the  actual  CPARS. 

We  also  noted  similar  issues  in  conducting  our  database  analysis,  that  in  some 
cases  the  exact  or  almost  the  exact  same  narrative  was  cut  and  pasted  into 
several  different  contract  report  cards. 

2.  Unsuccessful  Contracts  Are  More  Reliable,  Robust,  and 
Comprehensive 

Unsuccessful  contracts  tend  to  have  more  reliable,  robust,  and 
comprehensive  past  performance  information  available  in  their  CPARS/PPIRS 
reports. 
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Interviewee  Number  3  stated  that  there  might  be  a  reason  why  the 
unsuccessful  contracts  were  typically  more  robust  and  comprehensive  than  the 
successful  ones: 

If  the  contractor  is  really  just  coasting  along  doing  everything  really, 
really  well,  I  think  a  lot  of  times  you  might  not  have  a  really  in  depth 
report  because  the  COR  says,  “hey,  everything’s  good.  I  don’t  have 
to  worry  about  anything.”  So,  they’re  just  going  to  kind  of  mark  it 
up,  “hey,  they’re  good  or  they’re  bad  or  whatever  on  everything.”  I 
think  the  ones  that  maybe  would  take  longer  is  when  you  have 
issues. 

Interviewee  Number  3  went  on  further  to  state  that  narratives  might  be  longer  for 
a  poor  performance  because 

they  know  that  those  are  going  to  get  ...  the  contractor  is  going  to 
be  able  to  review  those  as  well  as  the  contracting  officer,  so  the 
COR  should  put  more  time  into  those  to  just,  you  know,  really 
explain  himself  why  you’re  giving  a  poor  rating.  Typically,  if  you’re 
getting  a  good  rating,  the  contractor  is  just  going  to  say,  “yup,  I 
agree.  Go  ahead  and  post  it  to  PPIRS.  I  agree  with  it.”  But  if 
you’re  giving  them  a  negative  rating,  you’re  really  going  to  want  to 
explain  yourself. 

3.  Assigning  Weight  to  Past  Performance  Using  the  Information 
Available 

The  appropriate  amount  of  weight  that  should  be  assigned  to  past 
performance  in  making  source  selection  decisions  should  be  correlated  to  the 
source,  availability,  quality,  and  relevancy  of  the  past  performance  information. 

Interviewee  Number  3  made  the  comment  that  the  weight  assigned  to 
contractor  past  performance  in  PPIRS  is  basically  a  direct  correlation  to  the 
relevancy  of  the  past  performance  combined  with  how  robust  and  comprehensive 
the  written  narrative  is. 

Interviewee  Number  5  stated  that  if  they  had  negative  past  performance 
information  from  PPIRS  or  from  a  questionnaire,  they  would  give  it  more  weight, 
but  generally  they  have  to  give  more  weight  to  the  technical  portions  of  a 
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proposal  because  PPIRS  is  not  able  to  produce  the  information  they  need  and 
the  questionnaire  is  not  likely  to  produce  negative  past  performance  information. 

Interviewee  Number  6  stated  that  it  is  more  beneficial  if  the  source 
selection  authority  is  able  to  access  multiple  contracts  of  similar  scope  and 
complexity.  Interviewee  Number  6  stated,  “If  there  is  consistency  I  would  place 
greater  value  in  that  evaluation.  If  they  are  all  over  the  place  then  it  is  really  hard 
to  put  a  ...  it  is  kind  of  subjective  how  much  value  you  put  into  a  particular 
CPAR.” 


4.  What  to  Do  With  Contradictory  Past  Performance  Information 

PPIRS  sometimes  contains  information  in  the  narrative  that  is  either 
contradictory  or  does  not  quite  match  up  with  the  objective  scores.  When  the 
objective  scores  and  narrative  sentiment  in  PPIRS  is  mismatched,  contracting 
professionals  tend  to  give  more  weight  to  the  narrative  versus  the  objective 
scores. 

Interviewee  Number  1  made  the  strong  recommendation  that  the 
contracting  professionals  who  regularly  deal  with  PPIRS  in  looking  for  past 
performance  information  need  to  take  note  of  when  there  is  conflicting 
information  between  the  objective  scores  assigned  and  the  written  narrative.  That 
interviewee  said  that  when  source  selection  team  members  find  something  like 
that,  they  should  send  the  report  back  to  whoever  entered  the  information  and 
make  them  correct  that  section.  This  comment  was  reiterated  by  Interviewee 
Number  4: 

I’d  send  it  back  and  tell  whoever  wrote  it  to  come  up  with  why  their 
scores  was  what  it  is  and  explain  to  them,  look.  I’m  looking  at  this 
and  I’m  reviewing  what  you  wrote,  but  it  doesn’t  justify  the 
outstanding  rating  or  the  poor  rating. 

Interviewee  Number  2  stated  that  they  give  considerably  more  weight  to 
the  narrative  versus  the  objective  scores  because 


sometimes  you  will  see  the  government  rated  them  very  good,  but 
then  the  description  you’re  reading  is  like  ...  it  doesn’t  match  ... 
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there’s  a  description  next  to  what  very  good  should  be  and  then 
what  they’re  saying  doesn’t  match  very  good. 


Interviewee  Number  2  also  stated  that  the  narrative  is  usually  more  beneficial 
because  sometimes 

there’s  a  bunch  of  different  categories  and  depending  on  what  the 
contract  is  for,  a  lot  of  them  may  be  N/A,  so  the  description  is  a  lot 
better  than  just  saying  like  ...  like  if  you’re  doing  a  firmed  fixed  price 
contract  but  for  some  reason  they  rate  cost  control  and  they  say, 
“excellent,”  well,  I  would  hope  so  because  we’re  only  paying  the 
firm  fixed  price. 

5.  Reliable  and  Accurate  Past  Performance  Retrieval  Methods 
Are  Needed 

Contractor  past  performance  reports  might  not  be  as  accessible  as  they 
should  be  in  PPIRS  because  contracting  professionals  are  not  always  applying 
due  diligence  in  identifying  the  appropriate  entity  using  identifying  information 
within  the  contractor’s  parent  organization.  Examples  of  such  identifying 
information  would  be  an  appropriate  cage  code  or  DUNS  (Dun  and  Bradstreet 
Data  Universal  Numbering  System)  number. 

Interviewee  Number  3  alluded  to  the  fact  that  it  might  be  more  beneficial  to 
have  the  PPIRS  database  set  up  to  more  easily  find  similar  contracts  for  a  given 
contractor.  Interviewee  Number  3  stated  that 

one  of  the  things  we’re  really  looking  for  when  we’re  doing  our 
source  selection  is  the  relevancy  of  past  performance,  so  not  just, 

“hey,  they  performed  well  on  a  bunch  of  contracts.”  That  matters, 
but  not  as  much  as  it  would  matter  if  they  performed  really  well  on  a 
similar  type  contract  to  the  one  that  we’re  looking  to  award.  So, 
you  know,  you  may  have  a  contract  for  a  completely  different 
service  that  this  company  did  and  they  got  really  awesome  ratings, 
that  gives  us  an  idea  that  they’re  able  to  perform  well  and  are 
probably  good  workers,  but  what  does  that  really  tell  us  about  how 
they’re  going  to  perform  this  contract  that  is  a  completely  different 
type  of  service.  So,  we  use  it,  but  I  would  say  we’re  going  to  ...  we 
have  other  methods  of  doing  past  performance  research  to  look  at 
their  relevancy  and,  you  know  ...  yeah,  their  relevancy  to  the 
contract  that  we’re  looking  to  award.  To  me  that’s  more  important. 

It’s  how  will  they  perform  on  a  similar  type  contract.  But  if  they’ve 
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never  done  this  type  of  work  before,  yeah,  then  it  would  be  weighed 
more  heavily  than  this  stuff  that  you’re  looking  at  in  PPIRS  because 
that’s  all  you  really  have  to  go  on.  So,  I  think  the  reports  are  good 
enough  to  explain,  hey,  this  is  the  type  of  work  that’s  being  done, 
so  you  can  really  make  a  good  comparison. 

We  noted  that  while  this  might  give  the  contracting  professionals 
information  to  use  in  weighing  past  performance,  the  information  provided  as 
references  from  contractors  would  almost  certainly  be  skewed  to  positive 
references.  A  contractor  will  most  likely  omit  a  recent  contract  reference  that 
they  know  they  performed  poorly  on.  This  is  really  where  the  process  of 
incorporating  past  performance  information  into  source  selection  is  failing  in  that 
PPIRS  should  be  the  avenue  for  source  selection  officials  to  find  negative  past 
performance  information  if  it  exists.  Too  often,  it  looks  like  data  in  the  system  is 
not  organized  well  enough.  The  way  the  data  is  organized  is  not  helping  source 
selection  officials  mine  the  database  to  uncover  this  recent  and  relevant  negative 
past  performance  information  as  well  as  it  should. 

When  asked  about  ideas  of  how  CPARS/PPIRS  could  be  improved. 
Interviewee  Number  9  had  the  recommendation  that  “there  could  be  something  in 
the  PPIRS  system  that  would  allow  you  ...  to  weed  through  the  relevancy 
aspect.” 

6.  Past  Performance  Questionnaires  Are  Inadequate 

Source  selection  officials  often  need  to  solicit  contractors  for  references  or 
ask  them  to  fill  out  a  past  performance  questionnaire  because  there  is  a  lack  of 
reliable,  robust,  and  comprehensive  past  performance  information  available  in 
PPIRS. 

Because  they  are  often  unable  to  find  relevant,  robust,  and 
comprehensive  information  about  a  particular  proposal  in  PPIRS,  Interviewee 
Number  3  stated. 

In  our  solicitations  or  our  RFPs,  when  we  put  those  out  then  we’ll 
ask  for  the  contractor  to  submit  ...  for  example,  we  might  ask  them 
to  submit  five  of  their  most  recent  contracts  that  they’ve  done  for 
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similar  type  work  with  references.  So,  they  would  provide 
references  of,  you  know,  John  Doe  who  they  did  work  for  in,  you 
know,  for  NAVSEA  out  of  Washington,  DC,  or  something  like  that.  I 
could  contact  him  and  say,  “hey,  how  did  these  guys  perform  on 
this?”  Or,  I  could  send  them  an  actual  ...  like  a  survey  to  fill  out  and 
then  we’ll  get  that  all  filled  out  and  then  we  know  that  it’s  a  relevant 
contract,  we  know  it’s  a  recent  contract  that  they’ve  done.  So,  it’s  a 
lot  easier  to  use  that  information  than  it  is  a  lot  of  times  in  PPIRS, 
because  if  you  go  into  PPIRS  it’s  like  the  most  recent  one  might  be 
like  2009  or  2010  and  it’s  like  a  completely  different  contract. 

Interviewee  Number  5  made  note  of  a  similar  questionnaire  and  reiterated 
the  issue  with  it  by  stating. 

When  we  have  source  selections  here  at  our  agency,  we  usually 
have  the  contractors  fill  out  a  past  performance  questionnaire.  It  is 
a  standard  questionnaire  that  we  send  out  for  each  offeror  and  a 
government  representative  has  to  fill  them  out  and  send  them  back 
directly  to  the  contracting  officer.  Now  those  are  a  little  bit  different 
in  that  it  is  not  CPARS,  but  they  do  have  to  reference  it  and  we  do 
validate  that  it  is  accurate.  But  I  mean  I  think  that  sometimes  you 
find  that  of  course  they  are  only  going  to  give  references  for  their 
best  clients  so  they  are  always  going  to  be  good  or  at  least 
satisfactory. 

7.  Timely  Reports  Must  Be  of  High  Quality  to  Be  Useful 

There  has  been  a  recent  increase  in  senior-level  interest  to  complete 
required  CPARS  reports  in  a  timely  manner  but  not  an  increased  interest  in  the 
quality  of  those  reports. 

Interviewee  Number  2  stated  that  the  narratives  are 

not  as  lengthy  as  they  could  be.  Like,  if  you  figure  we  spend  three 
months  on  something,  you  would  think  that  they  would  have  like  a 
real  good  paragraph  ...  very  detailed — but  sometimes  it  seems  like 
I  get  more  information  talking  to  people  than  the  two  sentences 
they  put  in  CPARS. 

Interviewee  Number  7  brought  up  a  noticeably  increased  emphasis  on 
timely  completion  of  the  report  card  but  said  it  was  just  as  important  to  have 

quality,  because  really  you  are  wasting  your  time  if  you  are  just 
putting  this  ...  you  are  wasting  everybody’s  time.  The  emphasis  is. 
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right  now  more  timeliness,  we  have  gotten  it  from  very  high  up  that 
they  want  these  done,  they  haven’t  really  said  done  well.  So  I  think 
after  we  get  our  hands  around  and  get  people  doing  it  within  the 
timeframe  they  are  supposed  to  be  doing  it,  then  we  would  like  to 
turn  to  the  quality  of  them.  I  know  some  of  the  field  activities  have 
already  taken  that  step.  I  heard  one  field  activity  actually  had  ... 
the  commanding  officer  went  through  every  single  CPAR  done  in 
the  last  year  and  pulled  some  of  them  back  and  said  that  he  wanted 
it  redone  because  it  didn’t  have  a  narrative  that  supported  the 
ratings  they  gave  them.  So  with  that,  if  you  are  getting  good, 
quality  CPARS,  I  think  it  is  a  very  valuable  tool  and  one  that  should 
be  used  in  source  selections  because  it  is  like  having — it  is  like  the 
Angie’s  List  for  the  federal  government  that  you  have  something 
that  you  have  feedback  on  customers. 

8.  Miscellaneous  Recommendations  for  Improvement 

Several  of  the  interviewees  made  recommendations  for  improving 
services  acquisition  related  to  contractor  past  performance  information.  The 
recommendations  were  to  add  more  analysis  tools  and  performance  metrics  in 
PPIRS,  better  monitor  the  workload  of  CORs,  better  train  the  workforce  on  writing 
CPARS  narratives,  and  better  broadcast  the  findings  of  CPARS  Program  Office 
audits. 


Interviewee  Number  6  recommended  that  the  program  office  could 

load  CPARS  up.  I  mean,  you  could  have  cost  analyses,  you  could 
have  trend  analyses,  and  you  could  have  performance  analyses.  I 
mean  did  they  perform,  how  well  did  they  perform,  did  they  make 
their  timelines,  was  their  product  accepted,  did  it  go  through 
developmental  tests,  operational  tests,  was  it  kicked  back? 

Interviewee  Number  7  mentioned  the  workload  of  the  CORs  as  a  potential 
factor  in  the  inadequacy  of  the  contract  report  cards,  stating, 

I  did  statistics  on  how  many  contracts  and  we  had  some  CORs  that 
were  definitely  way  overloaded.  Based  on  what  they  were  saying, 
really  to  do  a  good  job  on  a  good  contract  you  should  only  be  a 
COR  on  like  one  major  contract.  Smaller  ones,  two  to  three,  but  we 
have  people — six  or  eight,  more  than  that.  So  our  workload  was — I 
don’t  think  that  was  the  reason  they  weren’t  doing  their  CPARS,  but 
it  was  one  of  the  reasons  they  weren’t  doing  all  of  the  things  they 
needed  to  do. 
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When  asked  about  ways  that  would  help  improve  CPARS  so  that  it  could 
be  more  useful,  Interviewee  Number  8  stated,  “Training  is  probably  the  biggest 
one.”  Interviewee  Number  8  also  stated,  “We  don’t  train  our  people  very  well  in 
writing,  especially  when  you  have  engineers  that  are  filling  out  some  of  these 
things.  They  can  be  rather  cryptic.” 

Interviewee  Number  8  also  made  the  recommendation  that  when  there  are 
audits  conducted  at  a  command  regarding  CPARS,  it  needs  to  be  better 
broadcast  to  the  people  involved  in  services  acquisition  to  better  gain  access  to 
the  results  of  those  audits  along  with  the  trends  across  the  whole  of  government. 
Specifically,  Interviewee  Number  8  stated, 

the  CPARS  Program  Office  goes  out  and  does  audits  periodically. 

There  really  hasn’t  been — one  time  they  did  a  lessons  learned  and 
publicized  that,  but  it  would  be  good  to  see  some  trends,  things  that 
they  could  suggest  that  they  are  seeing  people  having  a  hard  time 
doing.  You  know,  that  would  help  improve  CPARS  as  a  role  if  they 
are  finding  some  people;  certain  contracts  are  harder  to  do.  I  don’t 
know.  Whatever  they  are  finding  in  their  audits,  we  never  get  the 
result. 

E.  SUMMARY 

This  chapter  revealed  the  results  of  the  research  that  examines  the  value 
of  CPARS  report  card  narratives  for  service  contracts  as  they  relate  to  their 
associated  objective  scores.  This  chapter  began  with  an  overview  of  the  data 
used  to  support  the  research  presented  in  this  report.  The  overview  first 
described  the  database  created  for  this  report  in  order  to  show  from  where  the 
results  of  the  statistical  analysis  originated.  The  overview  then  provided  a 
description  of  the  series  of  interviews  that  were  conducted  to  show  from  where 
the  results  of  the  interview  findings  came.  Once  the  database  and  the  interviews 
were  described,  the  chapter  explained  the  statistical  analysis  findings  in  detail. 
Following  the  statistical  analysis  findings,  this  chapter  described  the  interview 
findings,  with  each  finding  supported  by  specific  commentary  from  the  interviews. 
The  next  chapter  includes  a  summary  of  the  research,  conclusion  of  the  findings, 
and  areas  for  further  research  to  enhance  the  knowledge  of  service  contracting. 
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V.  SUMMARY,  CONCLUSION,  AND  AREAS  FOR  FURTHER 

RESEARCH 


A.  SUMMARY 

The  DOD  obligated  approximately  $360  billion  in  fiscal  year  2012  for 
contracts  for  supplies  and  services  (GAO,  2013).  In  order  to  support  best 
practices  for  government  acquisition,  the  Office  of  Federal  Procurement  Policy 
(OFPP)  directs  in  FAR  Part  42  that  federal  contracting  agencies  shall  use 
CPARS  as  their  primary  tool  for  documenting  contractor  past  performance. 

CPARS  is  a  system  that  was  designed  to  be  used  by  contracting 
professionals  for  submitting  and  retrieving  contractor  past  performance 
information.  There  have  been  many  discrepancies  and  failures  specified  in 
recent  GAO  reports  with  the  utilization  of  CPARS.  Government  contracting 
agencies  are  failing  to  properly  document  contractor  past  performance 
information  within  the  schedule  requirements  mandated  by  OFPP.  In  a  2009 
report,  the  GAO  analyzed  data  from  2007  and  showed  that  DOD  contracting 
components  completed  required  contractor  past  performance  report  cards  less 
than  half  of  the  time  (GAO,  2009). 

Subsequently,  the  DOD  increased  its  focus  on  training  and  education  for 
contracting  professionals,  which  has  led  to  an  increase  in  contractor  performance 
assessments  being  completed  and  submitted  to  PPIRS.  In  the  last  published 
report  during  the  research  period  of  this  project,  the  GAO  (2013)  noted  significant 
gains  in  completion  rates.  Fifty-six  percent  of  required  reports  were  completed  in 
2011  while  74%  were  completed  in  2013.  However,  according  to  the  same 
report,  over  half  of  these  reports  were  submitted  late.  In  addition,  many  CPARS 
report  cards  contain  narratives  that  are  either  insufficiently  detailed  or  conflict 
with  their  associated  objective  scores.  Late  reports  lacking  sufficient  accurate 
information  provide  less-than-optimal  information  to  the  contracting  professionals 
who  rely  on  these  report  cards  for  source  selection  and  contract  administration 
purposes. 
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B.  CONCLUSION 


1.  Research  Findings 

The  purpose  of  this  research  was  to  determine  the  value  of  contractor 
performance  assessment  report  narratives  for  services  contracts  by  comparing 
the  relationships  between  narratives  and  objective  scores  in  order  to  recommend 
improvements  to  the  CPARS  contractor  performance  information  documentation 
process.  To  identify  this  value,  the  research  focused  on  answering  five  research 
questions.  Our  research  questions  and  findings  are  as  follows. 

•  To  what  degree  are  government  contracting  professionals 
submitting  contractor  performance  narratives  to  CPARS  in 
accordance  with  the  guidelines  provided  in  the  CPARS  user’s 
manual? 

To  answer  the  first  research  question,  the  data  indicated  that,  overall,  contracting 
professionals  inputting  reports  into  CPARS  are  doing  a  highly  effective  job  at 
ensuring  that  their  narratives  are  free  of  statements  to  avoid  (-96.9%).  These 
professionals  are  not  doing  as  well  in  addressing  all  performance  areas 
assessed  in  their  objective  scores  (-81.5%)  or  in  ensuring  that  the  narratives  are 
based  on  objective  data  (-77.5%).  The  areas  that  contracting  professionals  seem 
to  have  the  most  issues  with,  however,  are  writing  a  comprehensive  narrative 
(-62.5%)  and  ensuring  that  a  contracting  layman  (such  as  someone  who  might 
need  to  access  CPARS  data  in  order  to  make  a  decision  about  a  future  contract 
award)  can  fully  understand  the  work  performed  (-64.5%).  With  the  exception  of 
the  narratives  being  free  of  statements  to  avoid,  contracting  professionals  were 
more  effective  in  all  categories  when  writing  narratives  for  unsuccessful  contracts 
than  with  successful  ones. 

•  What  is  the  added  value  of  the  contractor  performance  narratives 
beyond  the  value  of  the  objective  scores  for  performance? 

In  answering  the  second  research  question,  the  data  indicated  that, 
overall,  contracting  professionals  inputting  data  into  CPARS  are  doing  a  better 
job  at  providing  beneficial  data  in  the  narrative  when  the  contract  is  unsuccessful 
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versus  when  it  is  successful.  Only  38.6%  of  the  sample  narratives,  whether 
successful  or  unsuccessful,  provided  a  very  good  or  exceptional  amount  of 
beneficial  data  above  and  beyond  what  could  be  gleaned  from  looking  over  the 
objective  scores  assigned.  This  shows  that  there  is  clearly  room  for  improvement 
in  this  area. 

•  What  is  the  statistical  relationship  between  the  sentiment  contained 
in  the  narratives  and  the  objective  scores  for  contractor 
evaluations? 

In  answering  the  third  research  question,  the  data  indicated  that,  overall, 
contracting  professionals  who  input  data  into  CPARS  are  writing  information  in 
their  narratives  that  contradicts  at  least  one  of  the  objective  scores  assigned 
-8.3%  of  the  time.  This  leaves  room  for  improvement.  Contracting  professionals 
were  slightly  better  at  matching  the  narrative  sentiment  to  the  objective  scores  in 
unsuccessful  contracts  (-81 .8%  scoring  either  very  good  or  exceptional)  than  in 
successful  contracts  (-63.2%  scoring  either  very  good  or  exceptional). 

•  To  what  degree  do  the  interview  findings  contradict,  support,  or 
enhance  the  findings  for  the  three  research  questions  above? 

In  answering  the  fourth  research  question,  the  interviews  indicated  that 
there  are  no  issues  with  narrative  statements  to  avoid.  This  supports  the 
database  analysis  conclusion  that  contracting  professionals  inputting  reports  into 
CPARS  are  highly  effective  at  ensuring  that  their  narratives  are  free  of 
statements  to  avoid  (-96.9%).  The  interviews  indicated  that  there  are  only  a  few 
minor  issues  with  addressing  all  performance  areas  assessed  and  ensuring  that 
the  narratives  are  based  on  objective  data.  This  supports  the  database  analysis 
conclusion  that  contracting  professionals  are  doing  a  decent  job  in  addressing  all 
performance  areas  assessed  in  their  objective  scores  (-81.5%)  or  in  ensuring 
that  the  narratives  are  based  on  objective  data  (-77.5%).  The  interviews 
indicated  that  there  are  significant  issues  with  writing  comprehensive  narratives. 
This  enhances  the  database  analysis  conclusion  that  contracting  professionals 
seem  to  have  significant  problems  with  writing  comprehensive  narratives 
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(-62.5%).  The  interviews  indicated  that  there  are  some  issues  with  writing 
narratives  that  ensure  a  contracting  layman  can  understand  the  work  performed, 
especially  with  engineering-specific  entries.  This  supports  the  database  analysis 
conclusion  that  contracting  professionals  seem  to  have  significant  problems  with 
ensuring  that  a  contracting  layman  (such  as  someone  who  might  need  to  access 
CPARS  data  in  order  to  make  a  decision  about  a  future  contract  award)  can  fully 
understand  the  work  performed  (-64.5%). 

•  What  conclusions  or  recommendations  can  we  draw  from  the 
answers  to  the  research  questions  above? 

Lastly,  in  answering  the  fifth  research  question,  we  can  conclude  that 
contracting  professionals  should  get  some  additional/remedial  training  on  writing 
robust  and  comprehensive  narratives  that  the  contracting  professionals  who  are 
accessing  these  reports  are  able  to  fully  understand.  It  may  also  be  beneficial  for 
contracting  professionals  to  get  additional  training  in  ensuring  that  the  narratives 
address  all  performance  areas  assessed  and  in  ensuring  that  the  narratives  are 
based  on  objective  data.  With  the  issues  outlined  in  this  section,  we  can  also 
recommend  that  quality  control  for  those  approving  the  narratives  posted  to 
CPARS,  as  well  as  those  using  the  data  from  PPIRS,  can  be  improved  by 
sending  unacceptable  reports  back  to  the  originator  for  correction.  We  conclude 
that  there  is  no  further  emphasis  or  training  needed  in  ensuring  that  the 
narratives  are  free  of  statements  to  avoid  since  the  data  does  not  show  that  this 
is  a  significant  issue. 

2.  Additional  Findings 

In  the  process  of  completing  our  research,  especially  in  conducting  our 
interviews,  we  found  several  other  pieces  of  information  that  could  be  beneficial 
to  the  acquisition  community  with  regard  to  services  acquisition  that  we  would 
like  to  present  here. 

Finding  1:  The  information  in  CPARS/PPIRS  is  oftentimes  not  reliable, 
robust,  or  comprehensive  enough  to  allow  source  selection  officials  to  place  a 
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significant  enough  weight  on  past  performance.  This  finding  was  also  noted  in 
several  GAO  and  OFPP  memorandums  mentioned  earlier  in  the  literature  review 
chapter  of  this  report.  Without  reliable,  robust,  and  comprehensive  information,  it 
is  not  possible  to  properly  utilize  CPARS/PPIRS  in  making  fair  and  accurate 
source  selection  decisions  based  on  past  performance. 

Finding  2:  Unsuccessful  contracts  tend  to  have  more  reliable,  robust,  and 
comprehensive  past  performance  information  available  in  their  CPARS/PPIRS 
reports.  Source  selection  decisions  should  be  based  on  reliable,  robust,  and 
comprehensive  past  performance  information  for  both  successful  and 
unsuccessful  contracts  in  order  to  make  fair  and  accurate  award  decisions  based 
on  past  performance. 

Finding  3:  The  appropriate  amount  of  weight  that  should  be  assigned  to 
past  performance  in  making  a  source  selection  should  be  correlated  to  the 
source,  availability,  quality,  and  relevancy  of  the  past  performance  information. 
Implementing  this  finding  will  be  difficult  because  the  task  of  assigning  weight  to 
past  performance  information  is  normally  completed  during  the  acquisition 
planning  phase  of  the  acquisition  process.  Contractor  past  performance  is 
normally  mined  later  in  the  process  after  the  acquisition  team  has  received 
proposals. 

Finding  4:  PPIRS  sometimes  contains  information  in  the  narratives  that  is 
either  contradictory  or  does  not  quite  match  up  with  the  objective  scores.  When 
the  objective  scores  and  narrative  sentiment  in  PPIRS  is  mismatched, 
contracting  professionals  tend  to  give  more  weight  to  the  narrative  versus  the 
objective  scores.  When  the  objective  scores  and  narrative  sentiment  in  PPIRS 
are  mismatched,  every  effort  should  be  made  to  send  unacceptable  reports  back 
to  the  originator  for  correction.  If  this  is  not  possible,  the  weight  assigned  to  past 
performance  in  the  source  selection  decision  should  be  reduced  to  reflect  the  fact 
that  there  is  a  contradiction  in  the  report,  thereby  reducing  its  reliability. 
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Finding  5:  Contractor  past  performance  reports  might  not  be  as 
accessible  as  they  should  be  in  PPIRS  because  contracting  professionals  are  not 
always  applying  due  diligence  in  identifying  the  appropriate  entity  using 
identifying  information  within  the  contractor’s  parent  organization.  Examples  of 
such  identifying  information  would  be  an  appropriate  cage  code  or  DUNS  (Dun 
and  Bradstreet  Data  Universal  Numbering  System)  number.  It  is  imperative  that 
source  selection  officials  are  able  to  access  the  correct  reports  for  a  particular 
contractor’s  past  performance  if  they  exist.  There  must  be  a  uniform  way  of 
inputting  and  accessing  the  correct  information  in  PPIRS  if  source  selection 
officials  are  to  make  fair  and  reliable  award  decisions  using  the  past  performance 
information  that  we  have  about  a  particular  contractor. 

Finding  6:  Source  selection  officials  often  need  to  solicit  contractors  for 
references  or  ask  them  to  fill  out  a  past  performance  questionnaire  because 
there  is  a  lack  of  reliable,  robust,  and  comprehensive  past  performance 
information  available  in  PPIRS.  The  fundamental  issue  with  using  past 
performance  information  about  a  contractor  that  is  provided  by  that  contractor  is 
that  it  will  almost  certainly  only  include  positive  references  and  omit  references 
where  the  contractor  did  not  perform  well.  When  this  is  the  only  past  performance 
information  available  to  source  selection  officials,  the  weight  assigned  to  past 
performance  in  the  evaluation  criteria  needs  to  be  reduced  to  reflect  the 
reduction  in  the  reliability  and  comprehensiveness  of  the  information. 

Finding  7:  There  has  been  a  recent  increase  in  senior-level  interest  to 
complete  required  CPARS  reports  in  a  timely  manner  but  not  an  increased 
interest  in  the  quality  of  those  reports.  The  focus  needs  to  be  on  both  the 
timeliness  and  quality  of  the  reports.  Without  quality  reports,  the  information  in 
the  report  will  not  be  able  to  be  assigned  much  weight  in  future  source  selection 
decisions  and  therefore  will  just  be  a  waste  of  time  for  everyone  involved  in  the 
process. 

Finding  8:  Several  of  the  interviewees  made  recommendations  for 
improving  services  acquisition  related  to  contractor  past  performance 
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information.  The  recommendations  were  to  add  more  analysis  tools  and 
performance  metrics  in  PPIRS,  better  monitor  the  workload  of  CORs,  better  train 
the  workforce  on  writing  CPARS  narratives,  and  better  broadcast  the  findings  of 
CPARS  Program  Office  audits.  With  improved  analysis  tools,  appropriate 
workloads  for  contracting  professionals  inputting  the  past  performance  into 
CPARS,  and  a  more  robust  avenue  for  correcting  workforce  problem  areas,  the 
past  performance  information  in  PPIRS  will  undoubtedly  improve  and  better 
assist  source  selection  officials  in  making  fair  and  reliable  award  decisions. 

3.  Recommendations 

Based  on  our  conclusions,  we  identified  the  following  eight 
recommendations. 

Recommendation  1:  Remedial  training  should  be  implemented  for  all 
members  of  the  acquisition  teams  that  input  and  utilize  past  performance 
information  in  CPARS  and  PPIRS.  Training  should  focus  on  the  following  areas 
related  to  writing  comprehensive  narratives:  ensuring  that  acquisition  team 
members  (i.e.,  someone  who  might  need  to  access  CPARS  data  in  order  to 
make  a  decision  about  a  future  contract  award)  can  fully  understand  the  work 
performed,  addressing  all  performance  areas  assessed  in  their  objective  scores, 
and  ensuring  that  the  narratives  are  based  on  objective  data. 

Recommendation  2:  There  needs  to  be  an  overall  push  for  higher  quality 
past  performance  report  submissions  in  CPARS/PPIRS  to  allow  acquisition 
teams  to  assign  weight  to  past  performance  in  source  selection  decisions  without 
having  to  worry  about  the  source,  availability,  quality,  and  relevancy  of  the  past 
performance  information. 

Recommendation  3:  An  analysis  should  be  performed  on  the 
CPARS/PPIRS  database  by  the  program  office  to  ensure  that  contracting 
professionals  are  in  fact  able  to  accurately  input  and  retrieve  the  past 
performance  information  on  the  correct  contractor  100%  of  the  time  that  the 
information  is  available. 
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Recommendation  4:  A  review  of  the  CPARS/PPIRS  database  by  the 
program  office  should  be  conducted  to  see  whether  additional  analysis  tools  can 
be  added  to  better  assist  contracting  professionals  in  identifying  past 
performance  trends  for  a  particular  contractor. 

Recommendation  5:  Senior  officials  at  contracting  commands  should 
emphasize  not  only  the  timeliness  of  CPARS  submissions  but  also  emphasize 
the  quality  of  those  submissions. 

Recommendation  6:  A  requirement  should  be  added  to  the  CPARS 
report  card  approval  and  posting  process  that  allows  for  customer  feedback  on 
contractor  performance.  Agency  approving  officials  should  be  encouraged  or 
required  to  collect  (either  directly  or  via  the  COR)  and  consider  input  from  the 
customer  regarding  contractor  performance.  This  will  encourage  the  submission 
of  more  accurate  and  robust  CPARS  report  cards. 

Recommendation  7:  A  review  of  the  workload  of  CORs  should  be 
conducted  at  contracting  commands  to  ensure  they  have  the  appropriate  amount 
of  time  to  dedicate  to  the  proper  submission  of  past  performance  information. 

Recommendation  8:  the  CPARS  Program  Office  should  look  at  ways  to 
better  broadcast  the  findings  of  their  audits  to  help  the  workforce  improve  over  time. 

C.  AREAS  FOR  FURTHER  RESEARCH 

Additional  research  would  be  valuable  in  several  areas  to  further 
determine  the  value  of  the  CPAR  narratives  for  the  acquisition  process. 
Obtaining  a  larger,  more  diverse,  and  more  recent  sample  database  and  redoing 
the  database  analysis  would  add  significant  insight  into  how  the  contract  report 
cards  for  other  types  of  contracts  and  for  different  uniformed  services  are 
impacting  the  acquisition  process.  Specifically,  obtaining  a  new  database  that 
contains  contracts  for  supplies  and  contracts  for  different  military  branches,  and 
comparing  CPARS  report  cards  from  different  agencies  would  allow  for  some 
interesting  comparisons  that  could  yield  significantly  different  recommendations 
for  improving  the  use  of  past  performance  information  in  the  acquisition  process. 
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APPENDIX  A.  RESULTS  OF  THE  INTER-RATER  RELIABILITY 
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APPENDIX  B.  SPECIFIC  EXAMPLES  OF  NARRATIVE 
STATEMENTS  TO  AVOID 


FROM  THE  CPAR  QUALITY  CHECKLIST _ 

Block  20:  Sample  Narrative  Statements  to  Avoid 

The  Contractor’s  performance  in  this  area  was  exemplary.  They  were  proactive  in  satisfying  Electrical  Kit 
Product  Performance  requirements.  They  produced  a  superior  product  for  the  customer.  In  many 
instances,  they  performed  engineering  tasks  outside  the  scope  of  the  contract. 

“Outside  the  scope  of  the  contract’’  -  This  phrase  should  not  be  in  a  CPAR  narrative.  It  implies  that  the 
Contractor  performed  work  not  legally  required  and  is  eligible  for  an  equitable  adjustment  to  the  contract.  An 
equitable  adjustment  means  that  the  program  office/customer  will  have  to  come  up  with  additional  funds  to 
pay  for  the  additional  tasks. 

In  our  opinion,  the  Contractor’s  performance  in  the  systems  engineering  area  was  very  poor.  Kit  hardware 
deficiencies  were  observed  and  it  appeared  that  the  Contractor  lacked  systems  engineering  knowledge  and 
expertise.  We  believe  that  some  of  the  contractual  kit  requirements  will  not  be  met.  It  is  our  hope  that 
additional  factory  testing  will  eliminate  these  hardware  deficiencies.  If  management  had  responded  in  a 
timely  manner,  the  requirement  might  have  been  satisfied.  Additionally,  we  were  not  happy  with  the  initial 
factory  testing,  and  did  not  like  their  “fly  and  fix”  philosophy  of  testing. 

“In  our  opinion”  -  This  is  a  subjective  phrase  which  gives  the  impression  that  there  is  no  firm  evidence  to 
prove  poor  performance. 

“Appeared”  -  This  is  a  speculative  remark  which  does  not  prove  that  they  lacked  systems  engineering 
knowledge. 

“We  believe”  -  This  is  also  a  speculative  remark.  It  does  not  prove  that  they  did  not  satisfy  some  kit 
requirements. 

“It  is  our  hope”  -  This  statement  does  not  belong  in  a  CPAR  narrative.  The  issue  is  whether  the  Contractor 
will  correct  the  deficiencies  using  factory  testing.  If  so,  the  narrative  should  indicate  the  pending  corrections. 
If  not,  justification  should  be  provided  as  to  why  the  factory  testing  failed  to  correct  the  problems. 

“We  were  not  happy”  -  This  is  an  emotional  and  subjective  statement  which  should  be  avoided.  The  CPAR 
should  reflect  justification  for  the  successes/failures  from  the  factory  test. 

“We  did  not  like”  -  The  customer  should  evaluate  the  results  of  the  fly  and  fix  tests  in  detail,  not  their  testing 
technique. 

The  Contractor  was  late  in  delivering  all  of  the  100  electric  kits.  We  think  that  one  reason  is  that  their 
systems  engineering  effort  was  poor  due  to  several  electrical  component  deficiencies.  Another  reason 
could  be  that  their  ability  to  manage  the  electrical  subcontracts  left  much  to  be  desired.  We  established  a  6 
month  extension  to  the  contract.  We  hope  they  can  deliver  the  100  kits  without  significant  discrepancies. 

“We  think”  -  This  phrase  implies  that  the  customer  has  not  proven  the  Contractor’s  poor  performance  with 
evidence. 

“Could  be”  -  This  phrase  indicates  that  the  customer  is  not  sure  that  the  reason  for  the  deficiencies  is  poor 
management.  There  is  no  proof  of  poor  management  here. 

“We  hope”  -  This  phrase  implies  that  the  delivery  of  the  kits  without  deficiencies  in  the  time  period  allotted  is 
a  desire,  not  a  contractual  requirement. 
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APPENDIX  C.  AN  EXAMPLE  OF  AN  INSUFFICIENT  NARRATIVE 
VERSUS.  A  BETTER  WAY  TO  WRITE  THIS 


TAKEN  FROM  THE  CPAR  QUALITY  CHECKLIST _ 

Block  20:  Sample  CPAR  Narrative 

Insufficient  Narrative: 

Schedule:  Very  Good  -  In  our  opinion,  the  Contractor  has  done  really  well  in  terms  of 
schedule.  The  Training  Manager,  Jack  Jones,  is  pleasant  and  easy  to  work  with.  He 
adapts  to  our  schedule  changes  amazingly  and  never  complains.  He  also  went  above 
and  beyond  and  fixed  our  printer  and  fax  without  charging  the  Government  and  he 
continued  to  meet  all  the  contract  objectives  in  the  interim.  Great  job! 

The  example  above  is  missing  detail  to  support  the  score  and  supporting  documentation 
and  metrics.  In  addition,  it  uses  an  individual’s  name,  addresses  work  outside  the 
contract  scope,  and  uses  subjective  phrases. 

A  Better  Way  to  Write  This: 

Schedule:  Very  Good  -  Contractor  successfully  executed  the  delivery  and  training 
requirements  for  this  period  ahead  of  schedule.  For  example,  there  were  20  training  site 
visits  scheduled  for  this  period  however,  the  Contractor  conducted  31  visits  in  the  same 
period  of  time.  The  Contractor  also  met  100%  of  the  13  contract  data  requirements  in  a 
45  day  timeframe  versus  the  60  days  allotted.  This  resulted  in  data  requirements  14-20 
being  completed  earlier  than  anticipated.  This  was  done  with  minimal  supervision  by  the 
program  office  hence  allowing  more  time  for  additional  projects.  A  20  site  preventative 
maintenance  visit  ran  behind  schedule  for  the  first  8  months  of  the  reporting  period  due  to 
equipment  failures,  but  Contractor  management  was  able  to  bring  the  visit  back  on 
schedule  due  to  implementation  of  an  aggressive  quality  management  system  and 
spares  availability  policy. 
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APPENDIX  D.  INTERVIEW  QUESTIONS  FOR  DOD 
CONTRACTING  PROFESSIONALS 


INTERVIEW  INTRODUCTION 

The  following  section  will  be  read  verbatim  to  each  interview  subject  in 
order  to  explain  to  them  the  purpose  of  the  interview  and  to  ask  their  permission 
to  perform  the  interview. 

Project  Title: 

Determining  the  Value  of  Performance  Evaluation  Narratives  for  Service 
Contracts 

Purpose  of  Interview: 

We  are  three  graduate  students  enrolled  in  the  Acquisition  and  Contract 
Management  Curriculum  at  the  Naval  Postgraduate  School.  After  graduating,  we 
will  be  working  in  the  contracting  field  for  the  Department  of  Defense.  We  are 
conducting  research  in  an  effort  to  determine  the  value  of  contractor  performance 
assessment  report  narratives  for  services  contracts.  In  Department  of  Defense, 
members  of  the  acquisition  team  submit  contractor  past  performance  information 
to  Contractor  Performance  Assessment  Reporting  System  (CPARS).  This 
information  includes  objective  (numeric)  scores  for  contractor  performance  in  five 
categories  as  well  as  a  written  narrative.  Contracting  professionals  and  source 
selection  team  members  are  able  to  retrieve  contractor  past  performance 
information  that  was  submitted  to  CPARS  through  the  Past  Performance 
Information  Retrieval  System  (PPIRS). 

We  have  studied  and  conducted  statistical  analysis  on  CPARS  contractor 
past  performance  narratives  and  how  they  relate  to  their  associated  objective 
scores.  As  part  of  this  research,  we  would  like  to  ask  for  your  assistance  by 
allowing  us  to  interview  you.  The  purpose  of  the  interview  is  to  obtain  data  about 
how  and  to  what  extent  you  use  the  objective  scores  and  narratives  when 
reviewing  CPARS  (or  PPIRS)  report  cards  for  source  selection  and  contract 
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administration.  This  interview  data,  combined  with  a  statistical  analysis  of 
narratives  and  objectives  scores,  should  allow  us  to  make  recommendations  for 
improving  services  acquisition  and  streamlining  the  way  contracting  professionals 
submit  to  and  utilize  these  systems.  Our  goal  is  to  make  these  programs  more 
useful  to  you,  the  contracting  professional,  as  well  as  easier  to  use,  if  there  is  any 
improvement  to  be  had. 

Your  command  and  department  head  have  invited  us  here,  upon  our 
request,  to  conduct  this  interview  and  you  have  been  selected  by  them  to  be  a 
potential  interviewee  based  on  your  role  and  experience  as  a  contracting 
professional.  Participation  in  the  interview  is  voluntary  and  anonymous.  Your 
responses  to  our  questions  will  not  be  able  to  be  traced  back  to  you  or  your 
command  and  they  will  not  be  reported  to  your  supervisors  or  chain-of-command. 
This  interview  consists  of  13  questions  and  should  take  about  30  minutes  to 
complete. 

Do  you  agree  to  assist  us  in  conducting  this  research  by  allowing  us  to 
interview  you? 

Goal  of  the  Interview: 

The  previous  section  Purpose  of  Interview  was  formulated  with  the 
following  goals  in  mind: 

•  Brevity.  We  do  not  want  the  subjects  to  lose  interest  before  the 
interview  begins. 

•  Clarity.  This  introduction  to  our  research  is  concise  and  detailed 
enough  to  be  readily  understood  by  a  contracting  professional  with 
enough  knowledge  of  the  subject  matter  to  be  useful  in  our 
research. 

•  Unbiased.  We  do  not  want  to  show  whether  we  believe  contractor 
past  performance  objective  scores  have  any  more  or  less  value 
than  the  associated  narratives.  We  do  not  want  to  specify  whether 
there  is  in  fact  any  way  to  improve  upon  the  CPARS/PPIRS 
programs.  In  an  effort  to  avoid  encouraging  a  self-serving  bias  in 
interview  subjects  we  do  not  want  to  specify  that  submitting 
contractor  past  performance  information  to  CPARS  is  directed  by 
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Department  of  Defense.  We  want  to  get  them  to  answer  questions 
honestly  without  concern  that  they  might  not  have  been  closely 
following  protocol. 

•  Encouraging.  We  want  to  identify  with  our  subjects  as  fellow 
government  contracting  professionals  which  a  mutual  interest  in 
improving  our  processes  and  systems.  We  want  to  encourage 
them  to  agree  to  the  interview  and  to  be  forthcoming  because  in 
doing  so  they  might  help  all  of  us  in  the  future. 

•  Non-retribution.  Subjects  should  be  clear  on  the  fact  that  there  will 
be  no  way  that  this  interview  can  come  back  to  hurt  them  and  that 
their  identities  will  be  completely  protected.  Interviews  are 
completely  voluntary  and  encouraged  by  the  interviewee’s 
command. 

INTERVIEW  QUESTIONS 

How  often  do  you  use  contractor  past  performance  information  for 
source  seiections? 

[Provide  Verbal  Categories  of  Options  for  Answers] 

This  question  allows  us  to  approximate  the  level  of  depth  of  familiarity  the 
subject  has  with  contractor  past  performance  information  in  general  and 
specifically  whether  he  or  she  participates  in  contract  source  selections.  This 
question  is  carefully  worded  in  order  to  avoid  asking  alternative  questions  such 
as:  How  often  do  you  access  PPIRS?  What  portion  of  the  time,  when 
conducting  source  selection,  do  you  use  contractor  past  performance 
information?  We  want  to  know  specifically  how  many  times  in  a  given  period 
they  use  contractor  past  performance  information  for  source  selection  in  order  to 
determine  the  subject’s  depth  of  experience  and  to  determine  to  what  extent 
contractor  past  performance  information  is  used  as  compared  to  how  frequently  it 
is  used  during  contract  administration. 

In  order  to  help  the  subject  understand  the  question  we  will  provide 
examples  of  potential  answers,  carefully  in  order  to  avoid  injecting  bias,  such  as: 
once  a  year,  or  several  times  a  day.  Using  examples  on  the  extreme  ends  of  the 
range  will  keep  from  planting  a  likely  answer  in  the  mind  of  the  subject. 
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How  often  do  you  use  contractor  past  performance  information  for 
contract  administration? 

[Provide  Verbal  Categories  of  Options  for  Answers] 

This  question  is  the  same  as  the  previous  question  but  for  contract 
administration  instead  of  source  seiection.  The  purposes  of  this  question  are  the 
same  as  with  the  previous  question  except  for  isoiating  contract  administration 
from  source  seiection.  We  do  not  beiieve  it  wouid  be  sufficient  to  ask  one 
question:  How  often  do  you  use  contractor  past  performance  information  for 
contract  administration  and/or  source  seiection?  For  the  purposes  of  the 
research  we  want  to  know  to  what  extent  it  is  used  for  each  specific  purpose. 

When  inputting  past  performance  data  into  CPARS,  what  is  the 
typicai  amount  of  time  spent  per  contract  gathering  information  and 
inputting  it  into  the  system? 

There  is  some  purposefui  ambiguity  in  this  question.  Some  contracting 
professionais  write  a  the  entire  narrative  for  one  CPARS  report  card  white  in 
other  cases,  especiaiiy  for  iarge  and  compiex  acquisition,  the  task  is  shared  by 
up  to  severai  members  of  the  acquisition  team.  We  want  to  aiiow  the  interview 
subject  to  specify  how  much  time  is  spent  per  contract  gathering  and  inputting 
information  whether  or  not  aii  of  the  effort  was  conducted  by  the  interviewee. 
This  may  require  that  the  subject  make  some  estimates  of  how  much  time  other 
individuais  are  spending  on  this  task. 

Does  this  amount  of  time  vary  greatly  or  is  it  usually  around  the 
typical  amount  of  time? 

The  purpose  of  this  question  is  to  estabiish  whether  there  is  a  iot  of 
variabiiity  in  the  amount  of  time  one  spends  putting  together  a  CPARS 
submission,  if  there  is  not  much  variabiiity  then  the  foiiow  up  question  wouid  be 
to  determine  whether  there  is  much  variabiiity  in  the  compiexity  of  the  contracts 
the  subject  handies,  if  there  is  a  iot  of  variabiiity,  we  want  to  determine  whether 
the  variabiiity  is  based  primariiy  on  the  size  and  compiexity  contracts.  We  ciarify 
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the  impact  of  the  data  from  the  previous  question  with  one  of  the  two  following 
questions. 

[ask  the  following  question  if  subject  responded  with  answer  more 
similar  to  “the  amount  of  time  is  usually  around  the  typical  amount  of 
time”] 

Are  the  contracts  you  write  or  administer  usuaily  relatively  similar  in 
size  and  complexity  or  do  they  tend  to  vary  greatly? 

[ask  the  following  question  if  subject  responded  with  answer  more 
similar  to  “the  amount  of  time  varies  greatly”] 

Do  you  typically  find  that  gathering  and  submitting  information  to 
CPARS  takes  more  time  for  larger  more  complex  contracts  or  do  you  find 
that  the  amount  of  time  varies  not  related  to  the  size  and  complexity  of  the 
contract? 

Subjects  will  answer  one  of  the  two  previous  questions. 

Typically,  what  portion  of  the  total  time  it  takes  to  prepare  and 
submit  a  CPARS  report  is  spent  preparing  and  writing  the  narrative  portion 
of  it? 

The  purpose  of  this  question  is  to  determine  what  portion  of  their  time  is 
spent  preparing  the  narrative  which  will  be  one  of  the  evaluation  factors 
considered  when  determining  the  relative  value  of  the  narrative.  Our  evaluation 
will  compare  the  relative  value  of  the  narratives  to  the  relative  amount  of  effort  it 
takes  to  create  them. 

What  is  the  typical  number  of  people  involved  per  contract  in 
entering  contractor  past  performance  information  into  CPARS? 

The  purpose  of  this  question  is  to  tell  us  the  typical  size  of  the  acquisition 
team.  This  provides  us  data  useful  in  analyzing  the  CPARS  written  narratives. 
Some  narratives  are  very  uniform  throughout  their  sections  in  terms  of  positive  or 
negative  sentiment,  while  others  have  much  more  variation.  Some  CPARS 
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written  narratives  are  written  by  one  individual  while  others  are  written  in  parts  by 
several.  Useful  data  will  show  whether  it  is  much  more  or  much  less  likely  that 
an  individual  written  narrative  \//as  written  by  one  person  than  more  than  one 
person. 

[Ask  the  following  question  if  the  answer  to  the  previous  question 
was  greater  than  one] 

Which  members  of  the  acquisition  team  (contracting  officer,  COR, 
program  manager,  requirements  manager,  etc.)  are  responsibie  for  entering 
past  performance  information  into  CPARS? 

The  purpose  of  this  question  is  to  determine  specifically  which  members  of 
the  acquisition  team  are  most  likely  to  participate  in  the  writing  of  a  CPARS 
narrative. 

When  you  are  conducting  a  source  selection  or  administering  a 
contract  does  contractor  past  performance  information  have  any  impact? 

The  purpose  of  this  question  is  to  determine  to  what  extent,  in  general,  the 
interview  subject  values  contractor  past  performance  information.  In  this  case, 
we  feel  that  combining  source  selection  and  contract  administration  will  provide 
adequately  specific  and  useful  data  for  our  research  purposes.  The  question  is 
specifically  worded  to  avoid  injecting  bias  and  will  apply  to  all  interview  subjects 
irrespective  of  answers  to  previous  interview  questions.  If  the  interview  subject 
answers  in  the  negative  then  there  is  no  need  for  a  follow-on  question. 

[If  answer  to  previous  question  was  in  the  affirmative,  ask  the 
following  question] 


To  what  extent  does  contractor  past  performance  information  impact 
your  source  selection  or  contract  administration? 

The  purpose  of  this  question  is  to  determine  the  value  of  CPARS/PPIRS 
to  the  individual  contracting  professional.  Contracting  officers  and  source 
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selection  teams  are  allowed  by  the  Federal  Acquisition  Regulation  (FAR)  to  use 
some  subjectivity  in  determining  how  much  to  consider  contractor  past 
performance.  Knowing  how  much  they  value  the  product  will  aid  us  in 
determining  whether  improvements  to  the  system  might  be  beneficial  and 
whether  the  quality  of  the  system  affects  how  much  value  is  assigned  to  it. 

Which  provides  more  impact,  the  objective  scores  or  the  narratives? 

The  purpose  of  this  question  is  to  compare  the  two  sections  of  the  CPARS 
Report  Card  in  terms  of  value  to  the  contracting  professional.  This  question  is 
carefully  worded  to  that  it  is  a  zero  sum  value  pool  split  between  the  two 
sections.  We  are  not  asking  how  much  total  value  each  section  provides 
independent  of  each  other. 

What  ideas  do  you  think  wouid  increase  the  impact  of  contractor 
past  performance  information  in  source  seiection  and/or  contract 
administration? 

The  purpose  of  this  question  is  to  get  opinions  from  the  interview  subject. 
Since  evaluating  past  performance  information  is  mandated  by  the  FAR,  our 
research  is  attempting  to  find  ways  in  which  CPARS/PPIRS  can  be  made  more 
user-friendly  and  responsive  for  this  purpose.  Opinions  from  experts  in  the  field 
will  be  very  useful. 

Do  you  feel  that  past  performance  information  is  weighted 
appropriately  in  source  selection? 

The  purpose  of  this  question  is  to  determine  whether  the  interview  subject 
believes  that  the  FAR  or  their  agency  puts  too  much  or  too  little  emphasis  on 
contractor  past  performance.  If  the  interview  subject  believes  there  is  too  much 
emphasis  on  past  performance,  it  might  be  because  the  CPARS/PPIRS  systems 
are  insufficient  for  providing  useful  information.  This  question  was  worded  to 
avoid  the  injection  of  bias.  The  data  obtained  from  the  answers  to  this  question 
will  be  particularly  useful  when  combined  with  the  information  from  the  other 
answers  received. 


81 


Do  you  find  that  the  narrative  usuaiiy  provides  usefui  information 
above  and  beyond  the  objective  scores? 

The  purpose  of  this  question  is  to  determine  whether  the  CPARS  narrative 
section  has  value  whether  or  not  the  objective  scores  have  more  value  than  the 
narrative.  It  is  possible  that  the  interview  subject  will  answer  yes  even  if  he  or 
she  answered  to  a  previous  question  that  the  objective  scores  are  more  impactful 
than  the  narrative. 

If  the  objective  scores  of  past  performance  information  were  to  be 
eiiminated,  how  wouid  that  impact  the  source  seiection  or  contract 
administration  process? 

The  purpose  of  this  question  is  to  establish  the  absolute  value  of  the 
CPARS  objective  scores.  It  allows  the  interview  subject  the  opportunity  to  give  a 
bottom  line  answer  to  the  question:  How  much  does  this  really  matter?  The 
answers  to  this  question  are  going  to  be  particularly  useful  when  compared  to 
answers  to  previous  interview  questions.  If  the  interviewee  finds  that  the 
narrative  section  is  much  more  impactful  than  the  objective  scores  then  the 
conclusion  will  be  far  different  than  if  the  opposite  is  true. 

If  the  narrative  portion  of  past  performance  information  were  to  be 
eiiminated,  how  wouid  that  impact  the  source  seiection  or  contract 
administration  process? 

The  purpose  of  this  question  is  the  same  as  the  previous  question  except 
for  the  narrative  portion  of  the  CPARS  Report  Card. 
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